About Simulations and Bloom's Learning Taxonomy 161 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 ABOUT SIMULATIONS AND BLOOM’S LEARNING TAXONOMY Craig Miller Normandale Community College Craig.Miller@Normandale.edu Nancy Nentl Nancy.Nentl@metrostate.edu Metropolitan State University Ruth Zietlow St. Cloud State University razietlow@stcloudstate.edu ABSTRACT This paper presents a discussion about Bloom’s Learning Taxonomy, a broadly accepted hierarchical model of learning that identifies various stages of learning, and electronic learning tools widely distributed in today’s educational marketplace. First, the six successive stages of Bloom’s are discussed within the context of a business classroom. Then we differentiate six types of electronic learning tools that are commonly available to business instructors such as animations, discreet scenarios, branching, “smart” calculators, deterministic simulations, and Monte Carlo simulations. For each example, we provide a product description, its strengths and weaknesses, and identified its best use in the classroom. We then compare this hierarchy of experiential learning tools to the stages of Bloom’s Learning hierarchy. The differences between true simulations and quasi simulations or products that are not simulations, are significant and important to distinguish. We suggest that while many products are called “simulations,” they are not, in the truest sense, simulations. Whether or not an experiential learning product is a true simulation is measured by the degree to which the product has external validity and thus truly represents a business environment. We conclude that applying the classification of simulation to electronic teaching tools that lack external validity and thus under represent or misrepresent today’s business environment can dilute critical analysis of its adequacy and accuracy. INTRODUCTION While the need for infusing sophisticated analysis and thought into a business curriculum is clear (Roy and Macchiette, 2005), bringing students to this level of thinking can be challenging. Inspiring critical thinking is vitally important for business students as they prepare for careers that require strategic analysis and decision making. Bloom’s Taxonomy of Learning Objectives, a classic learning model, is an important pedagogical framework for identification and assessment of higher learning, as it differentiates the most elementary learning levels from the highest levels of learning, and is useful in the business school setting. Moving business learning beyond the elementary stages, what Bloom calls knowledge, comprehension, and application, into the higher levels of learning such as analysis, synthesis and evaluation, should be central for business instructors. Of course, students need to know definitions, and understand concepts and applications, but it can’t stop there. Business students need to understand cross functional interactions and interrelatedness, the vicissitudes of business, and the impact of decision making on key areas of an organization. Using typical classroom teaching protocols and tools ranging from flash cards, quizzes, end-of-chapter cases, and similar activities can be and are used to measure mastery of topics, however, many of these tools are limited in their ability to elevate learning amd thinking into true critical analysis. We propose that there is, however, one type of learning tool that can be very effective in moving students from the lower rungs of learning to the upper rungs where true critical analysis and understanding takes place, and that is the simulation. However, only certain types of simulations have this capability. The purpose of this paper is to differentiate simulations and simulation-related products in a market saturated with educational products and tools designed to enhance learning and tie them to the learning hierarchy outlined in Bloom’s Taxonomy. 162 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 BLOOM’S TAXONOMY Benjamin Bloom’s Taxonomy of Educational Objectives, herein referred to as Bloom's Taxonomy, was developed at the University of Chicago by a group of cognitive psychologists and spearheaded by educational psychologist, Benjamin Bloom. The group’s intent to identify and standardize learning objectives for student achievement led to the collaboration of banks of test items, each measuring the same educational objective (Bloom, 1956; Krathwohl, 2002). Bloom’s Taxonomy rose to significant prominence in the 1960s with the increased emphasis on education during the period of Lyndon Baines Johnson’s Great Society, and today, the model is well known to many educators. Although Bloom’s taxonomical theory addresses three domains of learning (the cognitive, psychomotor, and affective), the primary interest of this paper is the cognitive domain with its six successive stages of learning: the lower- order learning of Knowledge, Comprehension, and Application; and the higher-order learning of Analysis, Synthesis, and Evaluation. The appeal of Bloom’s learning model is its elegance, simplicity, and versatility. Its elegance and simplicity emanate from the notion that learning presumes to occur in this linear and hierarchical fashion. That is, relatively simplistic learning such as concrete knowledge, comprehension and application must necessarily occur before learners can engage in more sophisticated and creative learning such as analysis, synthesis and evaluation (see Figure 1). The versatility of the model is seen in its application for all levels of education, from kindergarten through higher education, as well as its cross-disciplinary use in business, social and other applied sciences (Athanassiou, Mcnett & Harvey, 2003; Bissel & Lemons, 2006; Blazelton, 2000; Buxkemper & Hartfiel, 2003). Although Bloom’s has been broadly accepted, the hierarchical and linear nature of the model does have its critics who argue that while learning is linear and hierarchical, it is also iterative and dynamic, particularly in the higher learning stages (Zohar & Dori, 2003). In other words, the cognitive action of analyzing new information, synthesizing new information with other information, and then evaluating the parts of the whole and creating new knowledge is ongoing and interrelated, rather than strictly hierarchical, as presented in Figure 2. BLOOM’S TAXONOMY IN THE CLASSROOM To operationalize these learning stages and objectives in a business course, Moore, G.S; Winograd, K. & Lange, D. (2001) provide definitions and related action verbs that characterize each successive stage. Then, how successful achievement of each stage might be demonstrated with specific examples of business course activities is added (Nentl & Zietlow, 2008). Knowledge: Definition: To define, memorize and remember previously learned material such as common terms, specific facts, and basic concepts. Related verbs: List, describe, identify, show, label, and quote, etc. Demonstration of achievement: A student can define and recall terms, dates, events, places, or describe subject matter. Examples of course activities: Define the GDP; list the four Ps of Marketing (Product, Price, Place, Promotion); identify the purpose of a balance sheet and label its components. Comprehension: Definition: The ability to grasp conceptual meaning. Bloom’s Taxonomy of Learning: Linear Model Figure 1 Analysis Application Evaluation Synthesis Comprehension Knowledge 163 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 Related verbs: Understand, discuss, estimate, compare, contrast, rank, recognize, report, or explain facts and principles. Demonstration of achievement: A student can correctly explain the history of an event, report on the status of an organization, or differentiate phenomena. Examples of course activities: Summarize the most important social factors. Application: Definition: The ability to use learned material in new and concrete situations; to demonstrate the accurate use of a concept or theory in a different context. Related verbs: apply, relate, demonstrate, illustrate, interpret, solve. Demonstration of achievement: A student can apply a theory in a practical context or recognize and then use the correct methods to solve problems. Examples of course activities: Calculate a breakeven point for manufacturing a new product; utilize Porter’s Five Forces as a framework for understanding the challenges of starting a new business. Analysis: Definition: The ability to break down a complex problem into different parts and to determine the relationships between those parts. Related verbs: analyze, appraise, criticize, differentiate, discriminate, distinguish, examine, and experiment. Demonstration of achievement: A student can explain why a particular solution process works to resolve a problem. Can see patterns underlying content; deconstruct the critical components of a framework. Examples of course activities: Discuss what customer lifetime value is and why it is a potent measure of profitability. Design a segmentation strategy for the U.S. Hispanic market; write a paper on how the current tax structure in the U.S. impacts taxpayers at various income levels. Analyze the implications of corporate authorship when used as a secondary source. Synthesis: Definition: The ability to put parts together to form a new whole. Related verbs: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize. Demonstration of achievement: A student can rearrange, reconstruct or combine parts of a process to form and utilize a new whole. Examples of course activities: Write a five-year plan for an IT department at a state university whose budget is constrained by the legislature. Design a prototype of a new consumer product, and conduct a beta test. Formulate a problem statement that reflects a variety of industry data from different secondary sources. Evaluation: Definition: The ability to judge the value of material for a given purpose based on definite set of criteria. Related verbs: persuade, appraise, judge, recommend, conclude. Demonstration of achievement: A student can create a variety of ways to solve the problem and then, based on established criteria, select the solution method best suited for the problem. Judge an argument’s veracity; evaluate another person’s work. Examples of course activities: Evaluate another student’s stock portfolio in a global industry. Assess the market value and profitability of a technological innovation. Write a case where ethical violations have occurred between two businesses and persuade the stakeholders to maintain the relationships in spite of the breach. Developing course activities and deliverables that reflect the lower learning stages of Bloom’s is relatively easy to accomplish for most instructors. In many curricula, Bloom’s Taxonomy of Learning: Nonlinear Model Figure 2 Knowledge Comprehension Application Analysis Synthesis Evaluation   164 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 in fact, there are courses whose primary learning objectives are to acquire the knowledge and comprehension of business fundamentals necessary for successive courses. These types of courses are essential and important because they provide the necessary foundation for students to advance to more substantive business courses. However, even more advanced courses inadvertently may have what amounts to lower order learning requisites, placing few demands on the students to unpack, add to, and evaluate existing knowledge structures. The challenge for instructors is moving students from the foundational stages of learning into the more dynamic interrelationship learning stages. Many types of course materials such as publisher test banks and end-of chapter exercises, etc. that are commonly used as course activities have a limited ability to engage in greater analytical thinking (Simkin & Kuechler, 2005; Davidson & Baldwin, 2005). However, some simulation products, depending on their design characteristics, can be effective in teaching students how to stretch to the upper bounds of learning as they assimilate, build on, and evaluate new and existing information (Miller & Nentl, 2001). WHAT IS A SIMULATION? A true simulation is an electronic model whose purpose is to reproduce an actual or theoretical reality. This type of simulation has been designed to represent any one of an array of environments such as engineering, medicine, the military, and manufacturing, and has a high level of external validity. Like other disciplines, a business simulation should also closely replicate the “real” corporate environment and teach students about the dynamics and cross functional impact of decision making (Miller & Nentl, 2001). A well designed simulation can teach students how to make strategic and tactical decisions in order to achieve stated business objectives, as well as long-term profitability, short-term revenue gains, operations, inventory levels, marketing and advertising, and market share. The key to a simulation’s ability to lift learning to this level of analysis and what distinguishes a true simulation from other types of simulations is the degree to which it has external validity. In other words, do the data points converge to form an accurate representation of a business environment? Does an action in one functional area simultaneously impact another functional area? Do actions and decisions have a long term impact of key business financials? Are the functions non-linear as they would be in business? External validity is derived from the internal design and architecture of the simulation itself. A simulation that has high external validity will integrate multifaceted architecture, designed around non-linear mathematics, simultaneous equations, and interdisciplinary dynamics such as finance, economics, marketing, and operations. The educational resource market is saturated with electronic learning tools, many of which have an appropriate place in teaching, but most of which are not at all accurate representations of the business environment. Some examples are animations, discrete scenarios, and branching decisions with linear outcomes, to name a few. While these types of tools play an important role in student learning, they all lack external validity. Thus, they do not accurately reflect the business environment, and have limited value for well informed discourse typical of the more analytical phases of Bloom’s. Following is a discussion of the characteristics of some of these tools and the level of learning in Bloom’s that is likely achieved. Then Tables 1 and 2 summarize this discussion, followed by examples, most of which were developed by one of the authors, Craig Miller (see Figure 3 in Appendix A). Animation: Not a simulation. Qualitative, non-mathematical, story- based, descriptive, visual, has motion. Platform: Typically Flash or PowerPoint™. Effective for demonstrating a step-by-step process or flow, demonstrating a concept, or idea. Examples: Penn’s Pens, an animation designed to teach the manufacturing process and corresponding accounting. MyLab, by Prentice Hall Bloom’s Taxonomy: Knowledge, Comprehension Discrete scenarios: Not a simulation. Primarily multiple choice questions with explanations related to textbook concepts. Platform: Typically Flash or PowerPoint™ Effective for definitions, testing recall, describing subject content. Examples: My Mentor, McGraw-Hill, designed to check comprehension of reading or lessons. Tend to be definition-driven. Bloom’s Taxonomy: Knowledge, Comprehension Branching Decisions: Not a true simulation, but can be called simulation-like. Scenario-based, decision driven with a finite number of decisions, take a linear path with several decision nodes. Based on if-then logic. Frequently designed as a game. Very limited external validity. Platform: Board games or computer games. Effective for teaching discrete cause-effect relationships. Examples: SimLife, a financial planning game. Bloom’s Taxonomy: Knowledge, Comprehension, Application, Analysis Smart Calculator: Quasi simulation. Standard financial “what-if” model, consisting of an input/output format. The input consists 165 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 of key business decisions such as financial allocations, price, budget considerations, marketing expenditures, etc. The output is generally in the form of financial measures such as income statements, cash flow and balance sheets. This type of learning object is not typically a game. No market responsiveness is integrated into this model. No external validity Platform: Spreadsheet technology such as Excel. Effective for demonstrating financial and accounting principles. Also can be used for teaching sensitivity analysis. Examples: MagFib, Dupont Chart used in Financial Courses Bloom’s Taxonomy: Knowledge, Comprehension, Application, Analysis Deterministic Simulation: A true simulation. A system of functions representing key business decisions such as price, promotion, marketing and advertising impact, productivity, operations, finance and accounting. The functions are interactive and non-linear using statistical methods that reflect true business, economic and marketplace dynamics. High external validity. Platform: Excel or other mathematical tools such as Mathamatica. Effective for demonstrating the systemic, cross functional nature of business. As a game, it can be played multiple times that lead to significant experiential learning. Example: SimSeries™ Bloom’s Taxonomy: Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation Monte Carlo: A true simulation. Similar to the Deterministic Simulation, except has a random variables component. Very high external validity. Platform: Excel or other mathematical tools such as Mathmatica. Effective as a powerful tool for business contingencies. Recommended for very experienced business leaders with an intimate understanding of market dynamics. Example: CapBud Monte Bloom’s Taxonomy: Synthesis, Evaluation. However, it is more of a planning tool than a teaching tool. CONCLUSION This paper seeks to tie Bloom’s Learning Taxonomy, a broadly accepted hierarchical model of learning that identifies various stages of learning, to the capabilities of learning tools widely used in today’s educational toolbox. Suggesting that the label “simulation” can be inaccurately applied or misunderstood, we have differentiated six forms of electronic learning tools that are broadly available in the educational marketplace today such as animations, discreet scenarios, branching, “smart” calculators, deterministic simulations, and Monte Carlo simulations. The differences between true simulations and quasi simulations or products that are no simulation at all, are significant and important to distinguish in terms of their features, robustness, external validity, and intended use. For each tool, we provided a product description, its strengths and weaknesses, and identified its best use in the classroom. In doing so, we created a hierarchy of experiential learning tools that is consistent with and tied to each step of Bloom’s Learning Teaching Tools Appropriate for Bloom’s Taxonomy Stages Table 1 Teaching Tool Knowledge Comprehension Application Analysis Synthesis Evaluation Animation √ √ Discrete scenarios √ √ √ Branching √ √ √ Smart Calculator √ √ √ √ Deterministic √ √ √ √ √ √ Monte Carlo √ √ √ √ √ √ 166 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 hierarchy. Applying the classification of simulation to electronic teaching tools that lack external validity and thus under represent or misrepresent today’s business environment can dilute critical analysis of its potency and can devalue the case for a true simulation. REFERENCES Athanassiou, N., Mcnett, J.M., & Harvey, C. (2003). Critical thinking in the managementclassroom: Bloom’s taxonomy as a learning tool. Journal of Management Education, 27(5), 533-333. Bissell, A.N. & Lemons, P.O. (2006). A new method for assessing critical thinking in the classroom. Bioscience, 56(1), 66-72. Blazelton, J.K. (2000). Students may blossom using Bloom's taxonomy in the accounting curriculum. In B.N. Schwartz & E. Kerz (Eds.), Advances in Accounting Education: Teaching and Curriculum Innovations, Vol. 2. Stamford, CT: JAI Press. Bloom, B. (1956). Taxonomy of Educational Objectives: the Classification of Educational Goals. New York: McKay. Buxkemper, A. & Hartfiel, D.J. (2003). Understanding. International Journal of Mathematical Education in Science and Technology, 34(6), 801-812. Davidson, R. & Baldwin, B. (2005). Cognitive skills objectives in intermediate accounting textbooks: Evidence from end-of-chapter material. Journal of Accounting Education, 23, 79-95. Krathwohl, David R. (Fall 2002). A revision of Bloom’s taxonomy: An Overview. Theory Into Practice, 41(4), 212-218. Miller, C. & Nentl, N. (2001). Learning by playing: Teaching business learners with computer simulations, Conference Proceedings Emerging Issues in Business and Technology. Myrtle Beach, NC. Moore, G.S., Winograd, K. & Lange, D. (2001). You can teach online: building a creative learning environment. Boston: McGraw-Hill. Nentl, N. J. & Zietlow, R.A. (2008). Using Bloom's taxonomy to teach critical thinking skills to business students. College and Undergraduate Libraries, 15(1- 2). Roy, A. & Macchiette, B. (2005). Debating the issues: A tool for augmenting critical thinking skills of marketing students. Journal of Marketing Education, 27, 264-276. Simkin, M.G. & Kuechler, W.L. (2005). Multiple-choice tests and student understanding: What is the connection? Decision Sciences Journal of Innovative Education, 3(1), 73-97. Zohar, A. & Dori, Y. (2003). Higher order thinking skills and low-achieving students: Are they mutually exclusive? Journal of Learning Sciences, 12(2), 145- 181. Teaching Tools and External Validity Table 2 Teaching Tool Has External Validity Is a True Simulation Animation Discrete scenarios Branching Smart Calculator ? Deterministic √ √ Monte Carlo √ √   167 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 APPENDIX A Examples of Electronic Teaching Tools Figure 3 Animation Example: Below is an electronic slide that is part of a presentation used to show beginning cost accounting students the flow of resources used in the manufacture of pens. It connects the physical flow of resources to cash flows, journal entries to the proper accounts and their eventual treatment on the income statement, balance sheet, and atement of cash flows. This is an effective tool for students that are visual / spatial learners that need to see the big picture. 168 Developments in Business Simulations and Experiential Learning, Volume 37, 2010     Discrete Scenario Example: This is a part of a series of short scenarios for students to practice using the correct time value of money computation. Students click the button identifying the correct answer. Each button has been programmed to provide feedback specific to that answer to help them understand the error. Clicking “D,” an incorrect response, will yield the following: “A CD is not a discounted instrument.” 169 Developments in Business Simulations and Experiential Learning, Volume 37, 2010     Financial Summary for Maria Cash and Checking 4,900$ Savings Account 3,000 Getting Started - (1) Select where you will live Current Payment Period Payment 1 Bedroom 600$ 1,800$ (2) Select your mode of transportation 2007 Toyota Corolla 1,000 1,020 341 (3) Do you want health insurance for the first year? Yes 400 (4) Do you want renters insurance for the first year? 30 Yes Continue to period 1Print Branching Example: Sometimes this type of tool is thought to be a simulation but with its very limited input variables (often a single node) and a finite and discreet set of outcomes (frequently only 2); it is not a “true simulation,” but rather a “quasi simulation.” The inherent linear nature of the model greatly limits the flexibility, robustness, and external validity. This type of interactive tool is effective when demonstrating how a single decision can affect subsequent options. The example below is from a 12-period game allowing students to explore the cumulative impact of their current choices such housing, transportation, insurance, entertainment, and others, has on personal liquidity. 170 Developments in Business Simulations and Experiential Learning, Volume 37, 2010     Smart Calculator Example: A classic tool used in beginning finance courses is the Dupont Chart, pictured below. It is a visual and mathematical representation of the components and relationships that determine Return on Equity. An advantage of the smart calculator is a developer can identify input or independent variables and create formulas for the dependent variables. In this example, the student can experiment with “what-if” scenarios and immediately see the outcome. This is a useful tool for case analysis and strategic planning. 171 Developments in Business Simulations and Experiential Learning, Volume 37, 2010     Deterministic Simulation Example: Below is an input screen for a decision period in a Marketing Simulation. This is different than the “smart calculator” in that market conditions, productivity and revenue functions are modeled using real data and curve fitting methods. These functions are typically non-linear and produce results that are subject to constraints and behaviors associated with the “real world.” Users must critically understand not only the relationship between an input variable (such as promotion dollars) and outcomes (such as unit sales), but synthesize and evaluate the relationship among input variables (price and promotion dollars) and output variable (sales units and profit margin for instance). Another difference is the underlying functions are not exposed to the user so student cannot try to solve them mathematically but rather through sound judgment and decision making. Table of Contents Volume 37, 2010 A Simulation in Organizational Behavior: Development and Beta Test Simulating Networks: An Experiment Pick Your Group Size: A Better Procedure to Resolve the Free-Rider Problem in a Business Simulation The Impact Of Playing A Marketing Simulation Game On Perceived Decision Making Ability Among Introductory Marketing Students If the Games Work, Why Aren't More Faculty Willing to Play? Modeling Cascading Demand: Accounting for the Effects of Captive Consumer Relationships Issues In Simulation Implementation: Lessons From a Freshman Seminar ENDURANCE: A game for current economic times Simulating Processes: An Application In Supply Chain Management Another Look At The Use Of Forecasting Accuracy On The Assessment Of Management Performance In Business Simulation Games The Effect Of Advertising On Demand In Business Simulations The Relationship Between Learning And Performance In A Total Enterprise Simulation: Revisited And New Data. Is the Gold /Pray Simulation Demand Model Valid and is It Really Robust? Mission Possible - Using Simulation Games For Management Training In A Transition Economy Business Game Modeling For The Negotiation Between General Contractor And Subcontractors The Relationship Between Goal Orientation And Simulation Performance With Attitude Change And Perceived Learning: A Follow-Up Study The Invalidity Of "Natural Market Structure" Pims Validation Of Marketing Games Examining A Beta Test An Experiment On Group Decision-Making Using A Business Game: An International Comparison Of Mba Students In Japan; China; And Russia A Platform for Business Games - From Game Playing to Game Making: The Case of Yokohama National University Checking Financial Balance of Target Brand Portfolio with the Strategic Market Plan Cash Flow Package About Simulations and Bloom's Learning Taxonomy Managing Complexity: Applying the Conscious-Competence Model to Experiential Learning Using Accumulated Profits to Assess Performance in Simulations And Now Let's Really Innovate: The Notion of Perpetual Online Courses Distance Learning: A Game Application Web Based Total Enterprise Simulation Learning Business Administration Using Simulation Software A Review of Experiential Exercises in Absel's Published Proceedings Moving Outside the Classroom: An Experiential Exercise Involving Video Clips Peer To Peer: A New Tool for Student Peer Evaluation Continuing To Assess Project Management: What's Been Done? What's It Means? Looking At New Twists Strategies for Promoting Knowledge Formation and Dissemination in Business Simulation and Experiential Learning Introducing Experiential Exercises To Inexperienced Faculty Members Conveying Concepts Through Cartoons: Preliminary Results From Software Engineers In An Industrial Organization Theory Class Utilizing Institutional Problems as an Experiential Learning Opportunity for Students Considerations in the Design of Vicarious Learning Environments Enhancing Critical Thinking Through Service-Learning As A Consultative Process Encouraging "Established" Faculty to Adopt a "New" Games and Simulations Pedagogy IMA Candy Company: An Experiential Exercise in Managerial Accounting Mustard Seeds as Means for Creative Problem Solving, Ethical Decision Making, Stakeholder Alliance, & Leader Development through Experiential Learning in Management Education Interactive Classroom Technology: Adding a Benefit to the Traditional Classroom Excelling in the Capstone Simulation: An Application of Spreadsheet Technology as a Decision Support Tool GEO: An Entrepreneurship-Oriented Computer-Assisted International Strategy Simulation ERP Simulation Game: A Distribution Game to Teach the Value of Integrated Systems