Learning Assurance Using Business Simulations: Applications to Executive Management Education Developments in Business Simulation and Experiential Learning, Volume 33, 2006 LEARNING ASSURANCE USING BUSINESS SIMULATIONS APPLICATIONS TO EXECUTIVE MANAGEMENT EDUCATION Owen P. Hall, Jr. Kenneth Ko Pepperdine University ABSTRACT Today the number of working managers returning to the classroom is growing rapidly as a result of globalization and technological developments. Many students are enrolling in executive management degree programs (EMBA) which feature flexibility and a focus on results. While most EMBA programs are delivered in a different format than the traditional MBA program the requirement for learning assurance is still essential. Simulations, which are used extensively throughout most EMBA programs as part of the experiential learning process, provide one approach for evaluating the effectiveness of curriculum design and delivery. The purpose of this paper is to illustrate how simulation in concert the development of rubrics can be used to support the learning assurance process in EMBA type programs. Keywords: Learning assurance, simulation, rubrics, executive management education. INTRODUCTION The number of working managers enrolling in executive MBA (EMBA) programs is growing (Edgington, 2004). EMBA programs are usually conducted in a style and format different from that of standard MBA programs (Fessler, 2001). Some specific characteristics unique to most EMBA programs include the following: • Reduces emphasis on traditional lecture format • Uses lock-step cohort student groups • Focus on collaboration and hands-on exercises • Caters to student work demands and travel schedules • Permits students to use actual work projects in courses • Features more learning from other students (andragogical) as compared with traditional MBA programs Nevertheless, the requirement for quality assurance is as essential as in the more traditional residential MBA programs. “Quality assurance” is defined as the process by which the educational institution measures learning outcomes against a set of specific goal and objectives. This process involves assessing content coverage, learning modalities, program rigor and resource support. One learning strategy that supports the need for a “quality assurance” is the Instructional Management System (IMS) cooperative initiative (Graves, 1999). This initiative is designed to promote systematic thinking regarding the delivery of higher education, to improve learning outcomes and to increase return on instruction investments. Specific principles of the IMS initiative include: 1) education involves more than a single course; 2) a course is more than content; 3) content is more important that lecture notes; 4) convenience is important, and 5) quality assurance requires an integrated learning approach. A core element in the quality assurance process is the formulation of learning goals. Some specific learning goals associated with many EMBA programs including the following: • Strategic perspectives – To integrate economic, social, technological and political trends into a holistic approach to business management. • Decision making –To develop cognitive competencies such as problem solving, critical thinking, making informed judgments, and using information efficiently. • Leadership – To inspire and work with others to achieve common goals. • Change management – To understand the benchmarking and best practice process. To formulate cost-effective plans having specific performance metrics. • Innovation – To foster an appreciation of the growing reliance on technology and how it can be used to enhance competitive advantage. • Globalization – To develop an international mindset including an awareness of different belief structures and cultural sensitivities. Each of these knowledge constructs is somewhat abstract in nature, especially when compared to more specific learning aims such as developing a cash flow analysis. Accordingly, these constructs are more difficult to measure and to assess (Drost, 2001). Furthermore, most EMBA programs do not use examinations or testing as a primary vehicle to evaluate student performance. Many EMBA programs are moving towards an increased focus on customization and experiential learning (Fry, 2002). These trends further complicate the assessment process. Thus the question is how can learning be measured and assessed for EMBA programs given these constraints? 1 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Figure 1 – Learning Assurance Process using Simulation Table 1 - Example EMBA Learning Goal & Objectives Goal: Students develop an integrated, multi-disciplinary, holistic approach to business management. Objective 1 Students’ recommendations and plans will incorporate relevant economic, social, technological and political trends that demonstrate a holistic awareness of contemporary business. Objective 2 Students will be able to accomplish the performance targets established the in business plan. Objective 3 Students will be able to develop and justify strategic recommendations that indicate the integration of a variety of business functions or disciplines. LEARNING ASSURANCE The AACSB “new” learning assurance philosophy can be characterized as follows (Miles, 2004): “Schools should assume great flexibility in fashioning curricula to meet their missions… Accreditation does not mandate any particular set of courses…Contents of the learning experiences provided by programs should be both current and relevant to needs of business and management positions.” Flexibility is a key ingredient in this “new” philosophy. Nevertheless, learning assessment does require a systematic evaluation approach. Shown in Figure 1 is the overall AACSB learning assurance scheme (Zhu, 2005). The basic approach consists of establishing basic learning goals, as outlined above, formulating precise objectives for each goal, defining specific rubrics (measurement categories) for each objective, assessing student performance against each rubric (in the figure simulation is used as the evaluation medium). The assessment outcomes then serve as a bases for revising the curriculum and providing faculty development as needed. The learning goals can also be modified based on the assessment findings. This final step provides for a self correcting feedback process. Table 1 presents an example goal and set of objectives for the strategic perspective learning aim outlined above. Similar goals and objectives would be developed for each of EMBA program learning targets. Two to four objectives would be assigned to each goal. A variation to the standard AACSB learning assurance process is Kirpatrick’s model (Galloway, 2005). Figure 2 presents Kirpatrick’s four element assessment system. The first step (reactions) in the Kirpatrick model involves obtaining feedback on student perspectives regarding the learning process generally via surveys. The next step is to assess specific learning outcomes using discipline specific mechanism (e.g., marketing simulation). The third step is to evaluate the student’s ability to apply course content to overall program goals. This evaluation can be done, for example, using a strategic simulation. The fourth and final step is to identify and measure post-graduation student achievement via surveys (e.g., salary). Thus the Kirpatrick model takes the learning assurance process into the marketplace as the ultimate assessment mechanism. 2 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 Figure 2 – Kirpatrick’s Model Structure RUBRICS DEVELOPMENT & ASSESSMENT Rubrics are at the heart of the assessment process (Ammons, 2005). A scoring rubric is a set of categories that is used to record the assessed performance for a given learning experience or assignment (Moskal, 2000). In an EMBA type program, a number of opportunities exist for assessing learning performance including: capstone experience (e.g., strategic simulation), course-embedded assessments (e.g., team presentations), portfolios (e.g., accumulation of student essays), andragogical centers (e.g., student lead instruction) and project thesis (e.g., business plan). Management integration is a core ingredient of most EMBA programs. This learning requirement is often addressed through the use of business simulations (McKone, 2003). Business simulations have also been found to be particularly effective in developing both individual and team management skills (Aguino, 2005). The development of team cohesion is also a typical objective of an EMBA program. Furthermore, evidence has shown that students engaged in simulations retain about 75% of the instructional content compared to 5% for lectures, 20% for audio-visual presentations, and 50% for discussion groups (Johne, 2003). A large number and variety of business simulations are presently available on the Internet (Cannon, 2005). An example assessment of several strategic simulations can be found in Appendix A. The flexible nature of Internet based simulations is particularly attractive for executive management programs. One specific operations type simulation that has found widespread use in EMBA-type programs is the Internet-based MIT Beer Game (Hong-Minh, 2000). This simulation is used to illustrate supply chain management principles in an interactive environment. Simulations provide the opportunity to measure student Table 2 – Rubric Template for Example Goal #1 Rubric Metric Poor Good Excellent 1 Simulation Business Plan Mission Statement + Goals 0 0 0 Overall Strategy 0 0 0 Tactical Analysis 0 0 0 2 Performance Outcomes Operations Targets 0 0 0 Marketing Targets 0 0 0 Financial Targets 0 0 0 3 Board of Directors Meeting Plan Rational 0 0 0 Presentation Quality 0 0 0 Q&A 0 0 0 3 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 learning both in terms of direct measures (e.g., earnings) as well as indirect measures (e.g., planning and team work). Table 2 presents an example scoring rubric template. The template identifies the learning goal and corresponding objectives. Associated with each objective is a rubric consisting of three elements, each one is measured on a three point scale. Other scales can also be used. In this illustration the assessments are based on both direct (Rubric 2) and indirect (Rubrics 1 & 3) evaluations made by the faculty. The context for this assessment is a strategic business simulation consisting of a number of competing firms (teams) with each firm composed of a number of students who each take on a specific managerial role (e.g., CEO). The focus of the first rubric is on the quality and completeness of the business plan. Specific assessment items would be the quality of the mission statement and the articulation of the business strategy (e.g., high volume, lose cost). The second rubric addresses performance outcomes of the business simulation. A typical business plan would identify specific operations, marketing and financial performance targets, e.g., increasing stock price by 10% on an annual basis. Most EMBA students already have experience in dealing with these types of business metrics which underscores the utility of using business simulations for assessing learning outcomes. The third rubric concentrates on the board of directors meeting (BDM) that would be held about halfway through the simulation. The supervising faculty would serve as the outside board members. Holding the BOD at the halfway point allows enough time for the teams to respond and to observe the consequences of any changes in the business model. The BDM presentation should provide the logic behind the current business plan and the rational for any proposed changes. The PowerPoint presentation should clearly identify the firm’s present status including specific challenges. All team members should participate in the presentation. The team should also show an ability “to think on their feet” to a variety of questions and issues raised by the board. Overal 22 16 9 Table 3 The three-po poor, 2 = go falling in t corrective ac learning assu “continuous The phi applied to b Table 4 illus example the abilities to p plan. One ap business pla bootcamp. F to have them Many plan c international One of simulation “loafing.” O “free-rider” individual pe and by havi assessments The use of s exclusively t 2003). Disci Table 4 – Assessment Response of G Goal Students develop an integrated, multi-disciplinary, making Rubrics Scoring Scores below expectations (score = 12) Specific Findings Students fail to appreciate the integrative nature of properly respond to changing simulation outcome Learning Strategies Increase focus on developing an integrated busines Faculty Development Participate in business plan competitions to better plan development in business Curriculum Design Provide a 1-unit course in business plan formulatio External Experiences Identify specific online resources and testimonials 4 Table 3 - Scoring Assessment l Score Assessment - 27 Exceeds Expectations - 21 Meets Expectations - 15 Below Expectations shows the overall scoring assessment scheme. int recording scale was converted based on: 1 = od and 3 = excellent. Typically, performance he “below expectations” category calls for tion. However, it should be pointed out that the rance paradigm is based on the principle of improvement”. losophy of continuous improvement can be oth the curriculum as well as at the faculty. trates this process for learning goal #1. In this focus is on improving the student teams’ repare and execute a strategic oriented business proach would be to provide a 1-unit course in n development or alternatively a business plan or the teaching faculty one palliative would be participate in a business plan competition. ompetitions are held annually both locally and ly. the challenges associated with using a team for rubrics assessment is the problem of ne approaching for addressing the “loafing” or problem is by having team members submit rformance assessments of each team member ng the faculty record individual performance based on specific tasks (Thavikulwat, 2004). imulation for rubrics assessment is not limited o capstone or team oriented events (Rommie, ple based simulation (e.g., marketing) can also oal #1 holistic approach to business decision- business management and fail to s s plan understand the role of planning and n regarding business plan development Developments in Business Simulation and Experiential Learning, Volume 33, 2006 be used to support the learning assurance process. The faculty assessment can be complemented by student feedback on individual team member contributions and personal experiences via web based surveys. Fessler, N., 2001, “Surviving an executive MBA riot,” Accounting Education, v10, i3, p325. Fry, B.; Carter, L., 2002, “Executive Education for state and local government: A Survey,” International Journal of Public Administration, v25, i8, p953. CONCLUSIONS Galloway, D., 2005, “Evaluating distance learning and e- learning: Is Kirpatrick’s model relevant?” Performance Improvement, v44, i4, p21. Executive management education is on the increase as a growing number of working managers are returning to the classroom. EMBA programs tend to be conducted in a different format than traditional MBA programs. Furthermore, executive management programs often place more emphasis on management integration, team building and experiential learning. “Quality assurance” is defined as the process by which the educational institution measures learning outcomes against a set of specific goal and objectives. AACSB “new” guidelines are extremely flexible and thus provide for a school specific assessment strategy. The overall assurance process consists of setting learning goals, establishing objectives and defining measurable rubrics. A scoring rubric is a set of categories that is used to record the assessed performance for a given learning experience or assignment. Simulations (strategic and discipline based) can be used as a vehicle for measuring learning outcomes using scoring rubrics. Simulations provide the opportunity to measure student learning both in terms of direct measures as well as indirect measures. The purpose of this paper is to illustrate how simulations can be used to support the learning assurance process in EMBA type programs. The approach outlined herein can also be applied to traditional MBA programs. Subsequent research will focus on the collection and analysis of data from faculty assessments, and student performance and feedback. Graves, W.H., 1999, “The instructional management system cooperative: converting random acts of progress into global progress,” Educom Review, v34, i6, p32. Hong-Mihn, S.; Disney, S.; Naim, M., 2000, “The dynamics of emergency transshipment supply chains,” International Journal of Physical Distribution & logistical Management, v30, i9, p788. Johne, M., 2003, “Virtual environments,” CMA Management, v76, i10, p28. McKone, K.; Bozewicz, J., 2003, The ISM simulation: Teaching integrated management concepts,” v27, i4, p497. Miles, M.; Hazeldine, M.; Munila, L., 2004, “The 2003 AACSB Accreditation Standards and implication for business faculty: A short note,” Journal of Education for Business, v80, i1, p29. Moskal, B., 2000, “Scoring rubrics: what, when and how?” Practical Assessment, Research & Evaluation, v7, i3. Rommie, G., 2003, “Learning outcomes of microworlds for management education,“ Management Learning, v34, i1, p51. Thavikulwat, P., 2004, “The tournament concept in assessment, “Simulation & Gaming, v35, i1, p5. Zhu, F.; McFarland, D., 2005, “Towards assurance of learning in business programs: Components and measurements,” Journal of American Academy, v7, i2, p68. REFERENCES Ammons, J.; Mills, S., 2005, “Course-embedded assessments for evaluating course-functional integration and improving the teaching-learning process,” Issues of Accounting Education, v20, i1 p1. Aquino, K.; Serva, M., 2005, “Using a dual role assessment to improve group dynamics and performance,” Journal of Management Education, v29, i1, p17. Cannon, H., 2005, “Simulation packages,” http://www.absel.org/Packages/packages.html Drost, D.; Chaney, J., 2001, “Core course outcome assessment and program continuous improvement using integrative business plan: an empirical evaluation,” Developments in Business Simulation and Experiential Learning, v28, p42. Edgington, R., 2004, Application Trends, Survey Executive Report 2004, Graduate Management Admission Council. 5 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 6 APPENDIX A Assessment Matrix of Selected Business Strategy Simulations (1-10, 10 = best), * Web-based Simulation Content (1) Interventions (2) Usability (3) Support (4) Cost (per student) Capstone* www.capsim.com/ 8 5 9 9 $40 Marketplace* www.marketplace-simulation.com/ 7 5 9 8 $45 MikesBikes* www.smartsims.com/ 7 7 9 8 $45 Markstrat www.boku.ac.at/iao/am/markstrat 7 5 8 6 $50 AGV* www.globalview.org/ 7 5 8 7 $60 Business Policy www.eskimo.com/~fritzsch/ 7 9 6 6 $20 1. Content – Extent of the coverage and integration of the basic management disciplines. 2. Interventions – Capability to inject ethical, economic, technological and human resource issues. 3. Usability – Ease of use for both students and instructors including quality of graphical user interface. 4. Support – Quality of the user’s manual and online resources including phone service. Table of Contents Volume 33, 2006 Learning Assurance Using Business Simulations Applications To Executive Management Education Team Teaching In An Integrated Business Course Using Critical Problem Based Learning Factors In An Integrated Undergraduate Business Curriculum: A Business Course Success Personality Type And Strategic Planning Business Games As Strategic Management Laboratories The Relationship Between Students' Success On A Simulation Exercise And Their Perception Of Its Effectiveness As A PBL Problem Forecasting Accuracy And Learning: The Key To Measuring Simulation Performance The Business Strategy Game: A Performance Review Of The New Online Edition Using The Socratic Method And Bloom's Taxonomy Of The Cognitive Domain To Enhance Online Discussion, Critical Thinking, And Student Learning Using Negotiation Exercises To Promote Critical Thinking Skills Effective Leadership Experiences For Management Majors In A Futures Class The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games Is Pay Inversion Ethical? A Three-Part Exercise Simulations And Experiential Exercises - Do They Result In Learning? Have We Figured It Out Yet? Examining Program Management In Business Simulations: Student And Faculty Views Validating Business Simulations: Do Simulations Exhibit Natural Market Structures? Characterizing Business Games Used In Distance Education Utilizing Games In A Graduate Level Instructional Game Course Employment Interview Preparation: Assessing The Writing-To-Learn Approach Simulations - Bridging From Thwarted Innovation To Disruptive Technology Creating An Authentic Cultural Lens Using Case Dialogue Learning By Fire: Reflections Of A First Time Online Instructor An International Internship With A Service-Learning Focus Learner Participation In The Online Learning Experience: Help Or Hindrance? Any Given Sunday: Intervention In Pursuit Of Simulation Team Parity Beginning With The End: Creating An Experiential Exercise From Assessment Criteria Simulating Life Cycles: Life Span As The Measure Of Performance In Business Gaming Simulations It's Puzzling: Communications, Competition, And Cooperation Balanced Scorecard Implementation For Strategy Management: Variation Of Manager Opinion In Real And Simulated Companies Cases And Business Games: The Perfect Match! Three-Attribute Interrelationships For Industry-Level Demand Equations Using A Web-Based Module To Teach Information Literacy Decision Support System For Demand Forecasting In Business Games The Invalidity Of Profit=F(Market Share) PIMS Validation Of Marketing Games Online Market Test Laboratory With The MINSIM* Program The Gas Mileage Game - A Policy Simulation Delivered Cost And Differentiation Applied To Threshold 3rd Ed. The Effect Of Team-Leadership Modes On Team Performance: A Preliminary Study The Design And Use Of A Macroeconomics Simulation Using Maple Software: A Pilot Study The Instructor's Toolbox: A Meaning-Centered Framework For The Social Construction Of Experiential Learning Incorporating Strategic Product-Mix Decisions Into Simulation Games: Modeling The 'Profitable-Product Death Spiral' Group Composition And Groupthink In A Business Game A Direct Approach To Teaching Business Ethics A Decision Support System For Planning Sales, Production, And Plant Addition With Manager: A Computer Simulation Polish - American Entrepreneurial Business Cooperation Workshop Utilizing The Income/Outcome Simulation Student Leader Training Exercise Student Preference To Mode Of Learning In Hong Kong Experiential Learning For Technology-Based And Management Programme In Hong Kong: A China Study Tour A Price Game With Product Differentiation In The Classroom Discrete Event Modeling In A New Transportation Simulation Supply-Side Modeling In A Total Enterprise Simulation The Quality Game Towards A Massive Multiplayer Online Business Simulation Making The Connection: Improving Virtual Team Performance Through Behavioral Assessment Profiling And Behavioral Cues Individual Learning Producing A Learning Organization: 'Playing Dice With Polar Bears' Narratology and Ludology: Competing Paradigms or Complementary Theories in Simulation Framework For Evaluating Internet Research Using Children's Games To Illustrate Strategy Concepts: Is Less Better?