ANALYZING MANAGERS' JUDGEMENTS AND DECISIONS WITH AN EDUCATIONAL BUSINESS SIMULATION Developments in Business Simulation and Experiential Learning, Volume 26, 1999 ANALYZING MANAGERS’ JUDGMENTS AND DECISIONS WITH AN EDUCATIONAL BUSINESS SIMULATION Gary J. Summers ABSRACT Business simulation has the potential for providing a student with cognitive analyses of his decisions and judgment. With this capability, a simulation can provide personalized training in critical thinking and business decision-making. Realizing this capability requires a simulation that (1) facilitates measuring information and knowledge and (2) relates the tasks demanded of the student to analyzable cognitive functions. This paper presents a simulation satisfying these requirements. It represents the first step in developing personalized, cognitive training in critical thinking and business decision-making. INTRODUCTION Currently business simulations teach through an indirect method. A student tries various strategies, analyzes the results, and, hopefully, infers an improved understanding. The student learns only as well as he can invent strategies and deduce lessons. This characteristic of business simulations might severely limit their effectiveness in teaching. In contrast, a direct method of teaching would analyze a student’s decisions and judgments in order to discover his unique, habitual judgment and decision strengths, errors, and biases. Such an analysis would not rely upon decision-making styles or personality types (e.g., Myers Briggs personality types). Rather, it would be cognitive. This cognitive analysis would facilitate personalized training in critical thinking and business decision-making. Potentially, business simulations are the ideal means of providing cognitive analysis and training. They present a student with well- defined problems and information which result in the receipt of well-defined answers. While necessary, these universal characteristics are insufficient. In order to provide personalized decision and judgment analysis, a business simulation must meet two additional requirements. First, the simulation design must facilitate measuring information and knowledge. Second, the business simulation must clearly relate the tasks demanded of students to cognitive functions that can be analyzed. This paper reports a preliminary effort at designing such a simulation. Specifically, the paper describes an application of the simulation design presented in Summers 1999 (published in this volume). This new simulation exhibits the two required qualities identified above. Being a preliminary investigation, this paper does not offer a protocol for cognitive analysis or a discussion of the relationship between research in cognitive psychology, management, and business simulation. Rather, it shows how the new simulation satisfies the two additional requirements and, in the process, illuminates the potential for cognitive analysis via business simulation. THE STUCTURE OF THE SIMULATION In its most basic form, the simulation proceeds in iterations of a three step process: students (1) analyze marketplace results, (2) design products, and (3) set production levels and send their production to the marketplace. In this process, product design is the fundamental task facing students. With this emphasis, it should not be surprising that the foundation of the simulation is its representation of products. The next two sections describe this foundation. The ensuing sections describe the three steps. Qualitatively Varying Products In this new simulation, products are composed of qualitatively varying attributes. Each attribute 58 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 expresses one characteristic from a set of characteristics. With this method, every product is represented with a unique vector of characteristics called a product’s design. Formally, let a product be composed of n attributes. A product’s design is specified by the vector where are the characteristics expressed by the attributes. ),...,,( 21 nccc nccc ,...,, 21 n A function V determines each product’s value. Products values are used to calculate demand using, for example, the Gold and Pray system of demand equations (Gold and Pray 1984). To understand the type of functions that can serve as the value function, one must consider the following question: “How much does a particular product characteristic contribute to a product’s value?” For example, how much value does a red exterior add to the value of an automobile? This question is difficult to answer. The value of a red exterior depends upon an automobile’s style. It is highly valued on sports cars but not on limousines. In this example, characteristics expressed by one attribute (style) affect the contribution to product value made by the characteristics expressed by another attribute (exterior color). This effect is called an interaction. Interactions make product design a difficult task because they create frustration. Frustration occurs when improving a product in one dimension decreases the product’s performance in other dimensions. ),...,,( 21 nccch= 1 The situation described above – attribute- characteristics representation and value functions with frustrating interactions – are characteristics of the problems encountered in the field of combinatorial optimization. Examples of such problems include designing the layout of an integrated circuit, finding the shortest tour connecting a set of cities, and finding a protein that catalyzes a particular reaction. The simulation described herein uses a combinatorial optimization function to assign values to products. 1 The value functions used in other simulations containing product attributes do not produce frustration (Teach 1990; Gold and Pray 1998). What kind of products could the system just described represent? Utilizing this system for ‘real’ products is problematic; one will have great difficulty in matching the combinatorial optimization function to a real product. Two methods resolve this dilemma. First, the product could be the object of a combinatorial optimization function (e.g., proteins or integrated circuits). Second, students could manufacture abstract products (e.g., a string of letters). Because students will have difficulty ‘feeling’ that they are managing a business when the product is abstract, one can give abstract products a visual representation, such as flowers or faces. For describing this simulation, the remainder of this paper assumes that products are strings of letters. Specifically, products are comprised of ten attributes, each expressing a letter of the alphabet. The vectors {QWERTYUIOP}, {ASDFGGFDSA}, and {CRDFBGFDSW} are examples of products. Analyzing Marketplace Results Students design products by selecting the characteristic expressed by each product attribute. To choose valuable product characteristics (the market favors high value products), students study the marketplace results from all previous rounds of play. Typical marketplace results are displayed in figure 1. The student knows which products sold; the round of the sales; the volume of the sales; and the manufacturing firm. By analyzing this information, the student identifies characteristics that contribute significantly to a product’s value. They identify these characteristics by their appearance in high value products and by their absence in low value products. For example, figure 1 suggests that products beginning with 59 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 ‘QWE’ are more valuable than products starting with ‘ACDF’. FIGURE 1 MARKETPLACE INFORMATION Column 1 2 3 4 5 6 7 8 9 10 11 12 13 Round Product Characteristics Firm Units Sold 1 Q W E R T Y U I U Y 1 25 1 Q W E B T N Y M I K 2 35 1 A C D F H N L I H F 3 12 1 A C D F N N R R G B 3 19 Designing Products Two difficulties confront students’ efforts to design high value products. First, there are an enormous number of designs. In our example, with ten attributes that each express one letter of the alphabet, students can choose from 26 products. A student can consider only a small number of these possibilities. Second, considering each attribute independently is an ineffective strategy. Frustration thwarts this potentially efficient strategy. 10 How can a student efficiently search for high value products? A student accomplishes this by hypothesizing a combination of characteristics that he believes comprises high value products. These combinations of characteristics define product classes (for example, black limousines or red sports cars). By restricting his search to product classes, a student can evaluate the potential of an entire class of products rather than evaluating every single product. He does this by evaluating a few products from a class and, using this information, estimating the class’s potential. If the product class shows potential, he concentrates his effort and investment in that class. If the class evaluates poorly, he hypothesizes new classes that he believes will produce better results. In order to develop valuable products more quickly and efficiently, the student will hypothesize several product classes and search within each one. The exploration of each hypothesized product class is called a project. In the simulation, a student will manage a portfolio of projects, deciding when to initiate new projects, when to cancel projects, and how to distribute his firm’s budget among projects. Figure 2 depicts a firm’s portfolio. In this figure, product classes are defined by listing the characteristics that define a class and placing a number sign in the remaining attributes. The number sign indicates that these attributes are not part of the class definition. For example, {ABC#######} represents the product class where the letters A, B, and C are expressed in the first, second, and third attributes, respectively. The products {ABCYHUKMNR} and {ABCRDWSZGY} are members of this class. FIGURE 2 A FIRM’S PORTFOLIO Projects #####C#BNT XYZ####### #AF#F###EB ABC####### Cash CowsNew Hypotheses Figure 2 shows four projects. Projects {#AF#F###EB} and {ABC#######} are cash cows. They produce products that are successful in the marketplace. The student managing this firm exploits these product classes through production. They provide his firm’s revenues. Although the student has two cash cows, competition compels him to search for higher value products. He must find these more quickly and efficiently than his competitors or suffer a competitive disadvantage. Product classes {#####C#BNT} and {XYZ#######} are the students hypotheses of product classes 60 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 containing higher value products. Production from these projects will likely be small as the student focuses on evaluation. Market Analysis Revisited: Perspective Students face a difficult problem in analyzing marketplace information. The marketplace produces an enormous amount of information. If attributes comprise products, the marketplace’s evaluation of a single product provides information about the value of 2 product classes. n n 2 No person can consider all of this information. To cope with the voluminous information, a student must select the information that is most effective and relevant to his business. He accomplishes this by evaluating only a few product classes. These classes might include, for example, the student’s projects, potential projects, and product classes defining his competitors’ products (as defined by the student). I call this set of product classes a student’s perspective. A perspective has the effect of categorizing the marketplace data. In doing so, it filters the market information, selecting the information that a student feels is most important. It is the means through which a student ‘frames’ the complex problem of competing, surviving, and profiting. Different perspectives filter the marketplace results differently. Students with different perspectives will identify and miss different opportunities; evaluate product classes differently; and value information differently. Results that are surprising to one student might easily be anticipated by a student with a different perspective. 2 A product presents a group of characteristics to the market. The number of combinations of characteristics evaluated by the market is the number of sets that one can create from objects. This number is n n n2 . Production Students make two manufacturing decisions each round: (1) purchasing and scrapping production capacity and (2) setting production volumes. Production capacity is purchased in ‘blocks’ (e.g., 100 units, 200 units, etc.) that are specific to a particular product (or a product class of small size). Producing other products incurs extra charges that increase the variable costs of production. For this reason, purchasing capacity is risky. A student should only purchase capacity when he has confidence in a product design. Cash cows should have dedicated capacity. Explorations of new product classes should incur the extra costs. To isolate this quality of the simulation, the price of production capacity and the price of raw materials are uniform over all products and kept constant throughout the simulation. MEASURING INFORMATION AND KNOWLEDGE Measuring Information In determining product designs and in hypothesizing product classes, students will use their knowledge of the value of products in a class to predict the values of products in other classes. To borrow a real world example, “What do strong sales of sports cars indicate about the values of utility vehicles?” The reliability of information measures this predictive capacity. The reliability of information is given by the correlation of product values between two product classes. The higher the correlation between two product classes, the better the values of the products in one class can predict the values of the products in the other class. One calculates this correlation by ‘pairing’ products from the two classes and applying the standard statistical formula. To pair products, one randomly selects a product from one class and then finds the product in the other class that is most similar to the selected product. Ties are broken randomly. 61 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 Measuring Knowledge: Core Competencies When searching a product class, a student learns the valuable combinations of characteristics for that class (the characteristics the ‘#’ attributes should express). With this knowledge, the student can efficiently improve his products’ designs. When this situation exists, the student has developed a core competency. The core competency embodies a student’s knowledge. One can record and the development of a student’s core competencies through statistical measures (measures of central tendency and variation) of the products that the student offers the market. For a set of products, one can measure a core competency with the vector },,,...,,{ 21 VVarAAA n nAAA ,...,, 21 Var . In this vector is an archetype product. Its characteristics are the characteristics represented most often in the set of products. Specifically, for the products in the set, is the characteristic expressed most often by the i iA th attribute. The variable Var measures the deviations of the actual products from the archetype. To describe this measure, it is useful to introduce the concept of Hamming distance. The Hamming distance between two products is equal to the number of characteristics by which the products differ. For example, the Hamming distance between products {QWERTYUIOP} and {QWERTYUMNB} is three. The variable is equal to the average of the Hamming distances between products in the set and the archetype product. The variable V represents the average value of the products in the set. One can apply this measure of core competency to any set of products (e.g., the products produced in a project, by a firm, or by all firms in a round of the simulation). By repeating this calculation over several rounds, one can track the evolution of core competencies. Figure 3 depicts this application. The horizontal axis indicates the round. The vertical axis indicates the number of characteristics by which an archetype product differs from the archetype product in round one. The figures progressing across the graph represent the core competency measure },,,...,,{ 21 VVarAAA n . The center of the figure represents the archetype. The span of the figures represents the variation in products, Var . Above each figure is the average value of the set of projects, V . FIGURE 3 TRACKING CORE COMPETENCIES 3.2 3.4 3.8 3.4 3.8 4.1 3.9 4.3 4.7 Value 0 10 1 2 3 4 5 6 7 8 9 Roun 1 2 3 4 5 6 7 8 9 d Figure 3 shows considerable movement in the development of a core competency. The large change in archetype between periods three and four suggests that the student has changed his focused to a new product class. The decrease in variation after round five indicates that the student has begun focusing upon production rather than search. A student would do this when he finds high value products. RELATING STUDENTS’ TASKS TO COGNITIVE FUNCTIONS In designing products, students are actually competing in solving a combinatorial optimization problem. Instead of using a scientist’s powerful mathematical algorithms, students use their own ‘cognitive algorithms’. In doing so, they exercise well-defined cognitive functions. Below, I describe some relationships between these cognitive functions and the tasks the simulation demands of students. Cognitive 62 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 psychologists have researched these cognitive functions. This research will provide guidance in developing a cognitive analysis of students’ decisions and judgments. Covariation Assessment When students analyze the marketplace data, they are searching for correlations between combinations of product characteristics and marketplace success. In cognitive and social psychology, this process is called covariation assessment. Experiments have tested peoples’ covariation assessment in a variety of situations. In one such experiment, subjects were shown several lists of paired variables and asked to estimate the correlation demonstrated in each list (Jennings, Amabile, and Ross 1982). This task is similar to students’ analysis of marketplace information. In this case, the paired variables are products and sales volume. The experiment shows dramatic results. Subjects’ estimates vary widely and, on average, greatly underestimate correlation. Correlations must be at least 0.8 before subjects, on average, estimate a correlation as high as 0.5. These results occur because subjects simplify their task by looking at only a few entries on the list. Correlation is a quality of the entire set, and only exceptional rows convey this quality accurately. This exercise suggests that when market results do not make facts obvious (market uncertainty), managers can be easily mislead by focusing their attention on a small set of information (e.g., the striking success, the striking failure, or firsthand experience). When market uncertainty exists, managers should rely more heavily upon decision rules and conduct a broad assessment of its industry. Categorization Categorization is a technique commonly used by people to simplify their environment. This is exactly what a student does when he hypothesizes product classes and a perspective (defining the market and his business). A student’s categorization will have a dramatic affect on his performance. To see this, suppose that each student associates each project within his portfolio with an estimate of its potential for producing profits. This estimate can be represented as a probability and updated each round. Different categorizations will incorporate marketplace results differently. Though they view the same information, the students’ expectations will evolve differently. Because of its strong influence on a student’s portfolio and project management, one should study how students form and change their categorization schemes. One potential analysis uses the mathematics of information theory. Given a student’s categorization and expectations, one can calculate the conditional entropy produced by the marketplace results. With this measure, one can calculate the amount of information available to a student as the industry evolves or as a student changes his perspective and portfolio of projects. From these studies, one can determine when students should, and when they do, change their market focus. Are students late in focusing upon new product classes? Analyzing a Student’s Judgment During the course of simulation a student must make the following project management judgments: the value of the products in a product class; the costs and time required to find valuable products; the reliability of information; and his level of confidence in his judgments. The student must also judge his portfolio’s risk, capital requirements, and potential for producing profitable returns. Each round of the simulation, one can solicit each of these judgments from a student. Furthermore, for each of these judgments the simulation can estimate the true value by sampling products and calculating correlations. From these values, the simulation administrator can identify which of the student’s judgments are 63 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 habitually erroneous. The administrator can also investigate how these errors affect a student’s project and portfolio management. With a suitable definition of risk, one can perform an analogous analysis of a student’s risk management. CONCLUSION Educational business simulations should be the ideal means of analyzing and teaching managerial decision-making and judgment. In order to accomplish this, a business simulation must (1) possess measures of information reliability and knowledge and (2) require a student to perform well defined, analyzable cognitive tasks. The simulation described in this paper satisfies these requirements by having students compete at solving a combinatorial optimization problem. This solution relates important business functions (project management and portfolio management) to measures of information and knowledge (reliability and core competencies) and to well-defined cognitive functions (correlation assessment, categorization, and judgment). This simulation provides a first step in building business simulations that provide personalized, cognitive analysis and training in managerial judgment and decision-making. REFERENCES Gold, S. & Pray, T. (1984) Modeling Market- and Firm-Level Demand Functions in Computerized Business Simulations. Simulation and Games, 15, 346 – 363. Gold, S. & Pray, T. (1998) Technological Change and Intertemporal Movements in Consumer Preferences in the Design of Computerized Business Simulations with Market Segmentation. Developments in Business Simulation and Experiential Learning, 25, 156 – 167. Jennings, D., Amabile, T., & Ross, L. (1982) Informal Covariation Assessment: Data-Based versus Theory-Based Judgments. In D. Kahneman, P. Slovic, & A. Tversky, eds., Judgment Under Uncertainty: Heuristics and Biases. New York: Cambridge University Press, 211 – 230. Summers, G. (1999) Modeling Innovation as a Process of Design in Educational Business Simulations. Developments in Business Simulation and Experiential Learning, 26 (this volume). Teach, R. (1990) Demand Equations for Business Simulations with Market Segments. Simulation and Gaming, 21, 423 – 442. 64 Table of Contents Volume 26, 1999 ABSEL's Historical Research Interests Back From the Future: An ABSEL Merlin Exercise for the Year 2005 The Contributions of ABSEL During the 1980's ABSEL's Contributions to Experiential Exercises in the 90's ABSEL's Contributions to Experiential learning/Experiential Exercises: The Decade of the 1970's Business Simulations - Algorithms and Model Enhancements: A 25 year Review A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning A Framework for Assessing the Competencies Reflected in Simulation Performance Developing A Learning Culture: Assessing Changes in Student Performance and Perception Financial Simulation Using Distributed Computing Technology Analyzing Managers' Judgements and Decisions with an Educational Business Simulation Understanding Your Business through Home-Made Simulator Development An Examination of a Reanalysis of the Impact of a Market Leader on Simulation Competitors' Strategies Applying Shocks to TE Simulations: A Demonstration Increasing Efficiency of Management Skill Assessment A Testbank for Measuring Total Enterprise Simulation Learning Developing Leadership Skills - Video Live! LEADSIMM: Collaborative Leadership Development for the Knowledge Society A Team Approach to Producing Multi-Media Laptop and Video Formatted Presentation Tinkertoys Revisited: Exploring Trust Based Relationships Overall Dominance in Total Enterprise Simulation Performance Success or Bankruptcy: The Relationship between Personal and Goal Orientation and Simulation Performance Assessing the Effects of Feedback: Muti-method and Muti-directions in Multi-pedagogical Courses The Missing Ingredients in Experiential Learning Purpose and Learning Benefits of Business Simulations: A Design and Development Perspective Building Capabilities for Change through Laboratory Simulations Modeling Innovation as a Process of Design in Educational Business Simulation The Need to Measurer variance in Experiential Learning and a New Statistic to do so Assessing Effectiveness of an Experiential Oriented Course Over Time Developing Participant Satisfaction Models of Experiential Exercises in Business Education Perceptions of Learning in TE Simulations Students' View on the Use of Business Gaming in Hong Kong Management Gaming's Lost Opportunity: Meditations about the Russian Experience and Prospects for the Future Unanticipated Enhancements in the Business Strategy and Policy Game when Running in Windows 95 Using a Business Game to Demonstrate Broad Business Concepts A New Model for Business Courses (Getting the Student Connected) Seeing the Forest and the Trees: Integrating Knowledge Using Large Scale Simulations in Capstone Business Strategy Classes Is It Here To Stay? A Roundtable Discussion of the Inter-Group Interaction Interactive Tools Used in Applying Financial Concepts Strengthening Essential Skills through a Finance Exercise: Calculation of Beta A Model of Currency Exchange Rates Understanding Currency Exchange Rates: A three-part Exercise Student Experiences in the World Intercollegiate Business Game Competition Student Experiences in the International-Collegiate Business Policy Game Competition Using Boards of Directors in Simulation Environments: Comments From Board Members Sharing Best Practices: Teaching Smarter, Not Harder Star Power: A Simulation for Understanding Power and Empowerment The Marketing Game: A Marketing Principles Simulation Alexander Islands: GSSM Tiny Business Simulator on the WWW Putting Strategy into Strategic Business Games Industry Analysis, Porter's Five Forces Model and Strategic Group Maps in the Business Strategy Game Simulation The Use of Concept Mapping to Improve Student Performance and Understanding of Strategic Management Concepts: A Comparison of Techniques Transformational thinking in the Organizational Behavior Course: The Use of Metaphor as an Assessment Tool Creation of a Virtual Learning Community for the Global Virtual Enterprise Project Inter Institutional use of Educational Resources: Joint Use of Management Simulation Games Business Insights: Theory and Practice with the Aid of a Business Simulation Progress: An Experiential Exercise in Development Marketing The ABC's of teaching the Theory of Constraints to Undergraduate Business Students Demonstrating Principles of Organizational Purchasing Behavior through an Experiential Exercise The Virtual Manager: A Different Simulation for Managing Complexity When the Rules are changing and Chaos Breeds Innovation: Recapturing the Value of Constructive Thinking and Play in Simulation Training The Pitfalls, and Potential, of Actual Events as Problem Drivers Using Business Games to teach Environmental Awareness and Green Management: The International Experience with the ENSIM Game A Review of my ABSEL-Related Work Simulation of Government: A Workshop Using GEO Creating Internet-Based Games Using Perl and JavaScript Who's on First? Exploring the Concepts of Problem-Based Learning, Experiential Learning, and Lifelong Learning Students Learn Customer Service and Selling while Conducting Research So You Want to Run an NFL Football Team–An Honors Interdisciplinary Project Supervised Internship: The Employer's Perspective Using Computer Assisted Simulation to Teach International Business Strategy: A Case of the Multinational Management Game (MMG) Multi-Cultural Experiential Learning: A Computer Simulation in China An Appreciative Stance on Diversity as We Move into the 21st Century: A Timeline Exercise to Identify Key Experiences in Good Work Relationships between Black and White Peers A Day in the Life of an Interactive, Real Time, and Internet Delivered Course: A Demonstration Comparing Internet Search Engines: An Experiential Learning Exercise Total Enterprise Simulations and the Internet: Assessing Student Perceptions and Preferences Teaching Accounting Information Systems in a Practicum Format Providing Experiential Learning in Accounting through a Field Study Payroll Project A Spreadsheet Based Business Simulation Game Computer-Behavioral Simulations Training for Project Managers Simulation Scenarios - Rationale and Illustration