EFFECTIVE USE OF MASTERY BASED EXPERIENTIAL LEARNING IN A PROJECT COURSE TO IMPROVE SKILLS IN SYSTEMS ANALYSIS AND DESIGN Developments In Business Simulation & Experiential Learning, Volume 24, 1997 104 EFFECTIVE USE OF MASTERY BASED EXPERIENTIAL LEARNING IN A PROJECT COURSE TO IMPROVE SKILLS IN SYSTEMS ANALYSIS AND DESIGN Arup K. Mukherjee. University of West Florida John L, Cox. University of West Florida ABSTRACT In this article we have reported about implementation of the ‘mastery learning’ pedagogy in a systems analysis and design project course. The approach has been used successfully for five years. As a consequence of use of this pedagogical approach, over 90% of students attained a pre- defined level of competence in significant techniques of analysis and design before passing the course. The most important drawback was about the large time commitment on the part of students and the instructor to make the process work. INTRODUCTION Experiential Learning is one of the most important modes of human learning from time immemorial. It has been suggested that the approach was first formally used by the Chinese over two thousand years ago when they used an examination for leadership in their national public service (Lorentz et al.. 1993). In modem times this mode of learning takes on a quite specific meaning in the context of an academic course. For example, according to Walter and Marks (1981), it involves a series of activities, which require active involvement of the students. Such an involvement may be in actual or simulated experiences. For example, apprenticeships internships, simulations, games, projects, case analyses and fieldwork are all examples of use of experiential learning. There is a very wide and deep literature about use of this approach in diverse academic fields like Management (Keys & Wolfe, 1990). Accounting (Specht & Sandlin. 1991). Marketing (Lorentz et al., 1993), and Psychology (Clements. 1995). For a brief but excellent overview of the approach refer to McKeachie (1994:). In the context of an undergraduate degree program in MIS/CIS many instructors use real life scenarios for developing important concepts. Programming problems assigned to students are based on problems or needs in different contexts. However, in the capstone systems project course, experiential learning is usually the critical, essential and most significant mode of learning. The systems project course has been recommended as a required course both in the DPMA and ACM model curriculum for CIS/MIS majors (DPMA. 1991). Over years, this course has become the capstone course for students in diverse degree programs like CIS MIS and software engineering. Most project courses pursue an implementation inclusive approach. This means that students are required to develop, test and debug a working prototype of an information system. This approach has many advantages. The students gain experience in developing a working prototype of a system with many interrelated parts, and in debugging the software. However, in our experience we observed many limitations to this implementation inclusive approach. The most important disadvantage was that in trying to meet the deadlines (e.g., end of semester, etc.) students were concentrating on putting together a prototype that was functional. They were paying little attention to other important aspects of the system. For example, they ignored design details like optimal access to information. Their databases lacked planning for efficient storage, retrieval and query-handling. It became clear that we were trying to do too much in too little a time. Our experience indicates that students in MIS/CIS programs usually take many courses where they Developments In Business Simulation & Experiential Learning, Volume 24, 1997 105 write computer programs. They also have experience in analysis and design of parts of small subsystems. However, there is no course where they think through the intricacies of a fairly complex system with many interdependent components as typically found in a real life scenario. Thus they are left with inadequate experience with tools needed to create a comprehensive model of an information system in a real setting. In other words these students are weaker in analysis and design in comparison to programming. Identification of this crucial weakness led to the design and development of an innovative capstone project course that did not require programming of the system. Instead, the course provides the students with a unique opportunity to focus on analysis and design of a complex information system in a real life setting. Details of the course are reported in Mukherjee (1996). Our implementation experience proved to be far from satisfactory. An important negative finding was that students exhibited a great degree of variation in the level of expertise in different aspects of analysis and design. In particular, major deficiencies were found in understanding of current system, identification of system components, construction of data flow diagrams, documentation of processing logic and design of data files. We found that most students performed poorly in one aspect of the project or another. While this list could go on and on the point is that student abilities in using techniques showed a wide degree of variation. All this implied that either their understanding of a technique or ability in executing a technique was poor. It was obvious that this weakness would carry over to their careers and lead to performance that was ‘unreliable’, unpredictable and possibly ‘substandard’. Clearly this was not a desirable state of affairs and we found it unacceptable. We felt that it was our responsibility to do something to ensure that students were let through the project course only after attaining a pre-defined level of competence in the execution of some of the important techniques of analysis and design. Such an approach has traditionally been known as mastery learning’. The basic idea behind mastery learning is that most students can attain a high level of capability in a subject matter if instruction is systematic, if needy students receive help when they fail to understand concepts or execute techniques, if they get sufficient time to achieve mastery and if they know what constitutes mastery (Bloom, 1968; Block, 1974). There is a large body of literature on the subject of mastery learning. For example, Guskey (1985) provides a good introduction to the subject and also lists over 100 references. In this paper we describe how we implemented the approaches of mastery learning in the systems design project course. Such an approach has been successfully used for five years at Fort Hays State University, Hays, Kansas, where the first author taught from 1990-1995. First we present a brief description of the systems design project course. Then we describe details of implementation of mastery learning strategies. This is followed by a discussion of benefits and limitations of such an approach for the systems project course. Finally, we summarize the findings and discuss issues pertaining to using such an approach. SYSTEMS DESIGN PROJECT The capstone project course is the 2nd course of a two-semester sequence. In the first semester the students are exposed to the concepts of analysis and design. In the second course they are required to apply these concepts and tools to a real life setting. The students work in-groups of two to four with three being the most common number. It is the group’s responsibility to identify a significant project in a real life setting. If they cant find one the instructor helps them identify one. Past projects have ranged in diversity and complexity. They have included such diverse applications as information systems for warehouse of international mail Developments In Business Simulation & Experiential Learning, Volume 24, 1997 sprinkler design, music scheduling in a radio station and course sequence advisor. During the semester, each group reports” progress on their project to the supervisor’ (instructor) in separate weekly meetings. The meetings usually last about 45 minutes to 1 hour and are used to clarify doubts, discuss problems, solution strategies and also ideas about applicability of different techniques. 106 Project Deliverables The major emphasis of the project is on analysis and design. The project is to be completed in distinct phases. A complete list of deliverables, broken down by phase is presented in the Appendix along with portions that are the responsibility of an individual member. In phases 4, 5 and 6 of the project the deliverables pertaining to the selected sub-system are the responsibility of the individual member. The course includes a presentation at the end of the semester. Each member must present a proportional share of the presentation. MASTERY BASED LEARNING In this section we describe the steps taken to implement the strategy of ‘mastery learning. The Mastery Process The mastery process consists of three distinct steps as shown in figure 1, and these steps are used in an iterative fashion in each phase as the teams work their way through the deliverables list. The output from each phase is included in a ring binder that also contains all their submissions pertaining to previous phases. This is necessary to help the instructor, as well as the student team members, recall details of the projects as well as the comments and suggestions made on previous submissions. In the first step, each group would submit a word- processed or computer generated submission of the deliverables for that particular phase. In the second step the instructor would grade and return the submission. This allows the instructor to point out mistakes, suggest modifications or indicate areas that need improvements. In the third step the group would make changes and re-submit. Steps two and three would be repeated until the group obtains a perfect score in that phase. Individual Effort In most phases there is a pre-specified amount of work that must be done by each individual member of a group. The individual member must perform the expected work, answer instructor’s questions about the work and be responsible for obtaining a perfect score for his/her portion of the phase. The Nature of Feedback An important element is the nature of instructor feedback on each submission. To encourage self- reliance and the work habit of attending to detail the practice that was implemented consisted of two Developments In Business Simulation & Experiential Learning, Volume 24, 1997 107 components. The first component of the feedback consisted of merely circling the area of the document where the error was made. The student had the important responsibility of discovering what the error was and following through with necessary corrections. The second component of the feedback consisted of writing down a few pointed’ questions with the intent of indicating deficiencies in the submission. The student had to make sure that the deficiencies indicated in the ‘pointed’ questions were satisfactorily taken care of in the new submission. Thus they learnt from their own mistakes and had a better understanding of ‘where’ and ‘how’ to use a particular technique. OBSERVATIONS ABOUT MASTERY LEARNING During five years of direct use of this mastery learning approach, we verified significant benefits and limitations, some of which are similar to those reported in the literature by other authors. The benefits, as well as the limitations, are discussed below. Benefits Several benefits are realized by use of this pedagogy in the project course. Some of the more important ones are summarized below: 1. The most important benefit is the achievement by the students of a predefined level of competence in some of the critical techniques of analysis and design before completion of the course. They quickly realize they have to become competent in the skills necessary in the profession. 2. The students gain an understanding of the level of quality that is likely to be expected of their work in the real world. They learn that production of quality work often requires many iterations. 3. At the start of the project, students usually have widely varying levels of expertise for the different tasks that they have to perform. As a result many mistakes are made by them. However, after the necessary attempts at re-working, they eventually attain good competence in the tasks and the degree of variation in expertise is reduced. 4. At the end of their efforts in the different phases they achieve perfect scores for the different tasks in the phases, which strengthens their self-esteem. 5. Most of the work needed extended periods of group discussion and group work. This improved their ability to work with others. Students learnt to ‘cooperate with group members and help weaker members. 6. The process of re-work requires students to find errors and deficiencies on their own. Thus they get to learn how to play devil’s advocate on their own work so that weaknesses are minimized. 7. In implementation of the pedagogy, the instructor is usually required to interact intensely with students who need help in understanding or applying concepts/techniques. The instructor gains an understanding of the student perspective on which techniques are difficult to use, under ‘what' circumstances and ‘why’. Limitations Mastery learning is not a panacea to educators. Some of the important limitations of use of this pedagogy in the project course are summarized below: 1. The students become competent in the techniques that they learnt using this pedagogy. This competence does not automatically extend to other techniques that are learnt using traditional pedagogies. 2. This pedagogical approach is extremely time consuming for students. Sometimes many iterations are needed to achieve high competence in a particular technique. It is necessary for the instructor to play the role of a coach and hold sessions to motivate the students at frequent intervals. 3. The approach demands great time commitment from the instructor. Grading submissions and resubmission’s, explaining techniques, identifying error patterns in individual students, identifying the real deficiency and working individually with particular students needs a great deal of time. A conservative estimate of the time needed is around three to five hours per group per week. This became a serious problem in spring 1995 when there were Developments In Business Simulation & Experiential Learning, Volume 24, 1997 108 7 groups requiring a total of around 20 to 25 hours of instructor time per week. 4. Achieving competence in use of techniques requires a lot of time. Hence the instructor has had to be selective and limit the number of techniques used in the course to a manageable few. SUMMARY AND CONCLUSION In this article we have reported about implementation of the mastery learning pedagogy in a systems analysis and design project course. The fundamental goal has been to ensure that students gain a predefined competence in significant techniques of analysis and design before passing this course. The most important advantage is that variation in student abilities in performing systems analysis and design is greatly reduced. The students gain the important insight that production of quality work often requires many iterations and a great deal of effort. The most important drawback is the large time commitment on the part of students and the instructor to make the process work. APPENDIX PROJECT DELIVERABLES Phase 1: Define the project Provide detailed description about the business and define the problem for which a computer-based solution is to be developed. Phase 2: Identify entities and attributes Phase 3: Develop decomposition/context/systems diagrams Identify sub-system that each member is responsible for. Phase 4: Identify transactions to be processes develop DFDs: document processes; design inputs and outputs; Phase 5: Identify all data stores for your sub-system: Develop logical DFDs: design files; Phase 6: Identify reports; document purpose of report: develop DFDs; design reports; Phase 7: Terminal dialogue Design flow charts for sequence and variations of screens: design screen layouts: Phase 8: Project presentation REFERENCES Block, J. H. (Ed.). (1974). Schools. Society and Mastery Learning. New York: Holt. Rinehart and Winston, Inc. Bloom, B. S. (1968). Learning for mastery. Evaluation Comment. 1:2 Clements, A. D. (1995) Experiential learning activities in undergraduate development psychology. Teaching of Psychology. 22:2, pp. 115-118. DPMA (1991) Information Systems: The DPMA model curriculum for a four-year undergraduate degree. Park Ridge, Illinois. Guskey, T. R. (1985) Implementing Mastery Learning. Belmont California: Wadsworth Publishing. Keys, B. & Wolfe, J. (1990). The role of management games and simulations in education and research Journal of Management Special issue 16:2. pp. 307-336. Lorentz R. L., Carland J. C., & Carland J. W. (1993). Learning by doing in marketing research: An experiment in experiential learning College Student Journal 27:3, pp. 284-287 McKeachie W. J. (1994) Teaching Tips. Strategies, Research and Theory for College and University Teachers 9th edition Lexington Massachusetts: D. C. Heath and Co. Mukherjee, A. K. (1996) A pedagogical alternative to implementation in the capstone project course for CIS/MIS majors. Submitted to the Journal of Computer Information Systems. Specht, L. B. & Sandlin P. K. (1991) The differential effects of experiential learning activities and traditional lecture classes in accounting. Simulation and Gaming 22:2, pp. 196-210 Walter, G. A. and Marks S. E. (1982) Experiential Learning and Change. New York: John Wiley and Sons. Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game