IS SIMULATION PERFORMANCE RELATED TO APPLICATION? AN EXPLORATION STUDY Developments in Business Simulation and Experiential Learning, Volume 29, 2002 IS SIMULATION PERFORMANCE RELATED TO APPLICATION? AN EXPLORATORY STUDY Anderson, Philip H. University of St Thomas PHAnderson@StThomas.edu Lawton, Leigh University of St Thomas L9Lawton@StThomas.edu ABSTRACT Questions about the efficacy of simulations as a pedagogy for learning about the application of course concepts was explored. Results of the test showed a significant relationship between the application of concepts presented in a basic marketing course and performance on a marketing simulation. The greater number of concepts that students utilized in the management of their simulation company, the higher their net sales revenues and net profits. Limitations and directions for future research are explored. INTRODUCTION Two key questions surrounding the use of simulations that have been troubled users since the very earliest days of gaming are: 1) What do participants learn from engaging in a simulation experience? 2) Is a simulation better than alternative pedagogies for accomplishing certain learning objectives? Presumably, in our role as business instructors, we attempt to provide students with a framework for viewing the discipline. The text, class materials, and our instruction generally are aimed at providing students with knowledge, concepts, prescriptive models, and skills that should lead them to improved decision-making abilities. While, as instructors, we hope that we are improving students’ decision-making skills, we usually have little direct evidence of such improvement. We often test students on their knowledge of the material, but are left to wonder if they are actually able to apply the concepts. Depending upon the nature of the course we teach and our own personal objectives, our inability to make an accurate assessment of a student’s ability to apply the concepts of our discipline may be more or less problematic. If we are teaching an overview course in which our major emphasis is to impart the terminology, the history, or the context of the discipline, we may not be particularly concerned with application. However, if one of our objectives involves application, we should be very concerned if we are unable to assess the capability of a student to implement the concepts of the course. As instructors, we should strive to select pedagogies that are consistent with the learning objectives of our courses (Gentry, et al., 1979). It seems intuitively reasonable, and there is evidence to support the belief that different pedagogies are better suited to accomplishing different aims. For example, Teach and Govahi (1988) conclude that, based on student perceptions, lectures are superior for learning to listen reflectively; cases are best for learning a set of nine skills including analyzing problems, conceptualizing, and writing effectively; and experiential exercises excel for a set of 17 skills. Other ABSEL researchers have found similar results (Miles, et al., 1986; Schellenberger, et al., 1989). In short, our selection of both the content of our courses and the processes we employ to teach that content is critical to our success in meeting our course objectives (Miller, H.E., et al., 1998). Those of us who use simulations as an integral pedagogy in our courses presumably believe that they facilitate learning. The nature of the learning that occurs from participating in a simulation has been a topic of considerable speculation (Burns, et al., 1990; Keys, 1976; Wolfe, 1990). Instructors have promoted simulations as a means for accomplishing a wide range of learning objectives. These have included improving interpersonal skills, improving general decision-making skills, and helping individuals understand themselves (Anderson and Lawton, 1997). Given this myriad of possible learning objectives, ABSEL researchers frequently have used Bloom’s Taxonomy as a framework for guiding their thinking on the areas where simulations are likely to have the greatest impact on learning (Bloom, et al., 1956). Bloom et al. (1956) developed a system for classifying learning into six levels arranged in hierarchical order to reflect progressively higher levels of learning. These levels are, in ascending order, basic knowledge, comprehension, 108 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 application, analysis, objective synthesis, and objective evaluation. While simulations undoubtedly have the potential to stimulate learning at all levels of Bloom’s hierarchy, most authors postulate that simulations are best suited to facilitate learning at the higher levels of learning (Anderson and Lawton, 1997; Gosen and Washbush (1993, 1994, 2000; Hemmasi and Graf, 1992). Since simulations require participants to act in the role of managers, it would seem likely that, if simulations excel in any areas, they would be strong in application (the discovery of relationships, generalization, and skills) and analysis (the solving of problems in light of conscious knowledge of relationships between components and the principles that organize the system). However, objective evidence for the simulation’s effectiveness at the higher levels of Bloom’s Taxonomy has been lacking. Whiteley and Faria, (1989) concluded that simulation games “are not an effective means by which to improve the acquisition of applied or theoretical knowledge” (pg. 78). Later, Anderson and Lawton (1997) reviewed the literature and reported “objective measures have been used for a very limited range of learning objectives” (pg. 68). They contend this is due to a lack of validated objective dependent measures of learning. More recently, Gosen and Washbush (1999) concluded, “perceived learning appears to be unrelated to objective indices of learning” (174). In sum, whether simulations are effective in imparting learning at the higher levels of Bloom’s hierarchy and whether simulations help to teach the application of the concepts of a discipline remain open questions. This paper focuses primarily on the first question raised above — What do participants learn from engaging in a simulation experience? If simulations are, indeed, helpful for teaching participants to apply the concepts of a discipline, how can we demonstrate this capability? To answer this question, we sought to explore whether those students who claim to utilize the knowledge and skills presented in class outperform those who fail to apply course concepts as a guide to decision-making in a simulation. We report the results of an exploratory study that examines the relationship between the application of principles of a discipline and performance on a simulation. RESEARCH METHODOLOGY Subjects for the study were juniors and seniors at a medium-sized, university located in the Midwest. All the students were traditional, college-aged students enrolled in a principles of marketing course. The course was required of all business majors so the students came from a variety of fields. A total of 27 students participated in the study. These students comprised nine simulation teams. The simulation was Threshold Marketer Solo, a new marketing game being by the authors. Threshold Marketer Solo is a moderately complex simulation requiring students to make approximately 70 decisions involving elements of the marketing mix (e.g., price, quality, promotion) for each period of play. There are also decisions relating to the purchase marketing research information and a small number of production and financial decisions. Each decision represents a three-month (i.e., one quarter) period. The simulation ran for eight periods. Students in the class were organized into three person teams and each student team competed against a set of eleven semi-intelligent computer-managed opponents. Although each student team operated in an industry entirely separate from each other student team, the parameters of each industry (the demand curves, the sales response functions, etc.) were set to be identical. Two dependent variables were employed in this study: 1) Net income for the second (final) year of the simulation, and 2) Total revenue for the second (final) year of the simulation. The second year was used (rather than the totals for the game) to give teams a chance to achieve a reasonable understanding of the simulation and to provide some time for their strategies to go take effect. Since each student team was competing in a separate industry against a set of computerized rivals, the net income of the student team as a proportion of the total industry-wide profits was used (rather than absolute net income). Using the proportion rather than the absolute dollar amount provided a better picture of how the student team fared in its industry. The same manipulation (proportion of the total industry-wide rather than absolute dollars) was made for total revenue. Appendix A shows a list we constructed to represent concepts covered in a Principles of Marketing course. The students in each student team were asked to indicate which of the listed concepts they attempted to use as a guide to their decision-making. The independent variable consisted of the number of marketing concepts that each student team reported attempting to use. The instrument was not administered until after the game was finished and the grades for the simulation had been assigned and students were assured that their answers would have no impact on their grade in the course. This was done in an effort to avoid the potential bias that could arise if students suspected that their grades might depend upon their answers. RESEARCH HYPOTHESES The premise underlying this research is that if simulation performance does reflect learning associated with analysis and application, those students who apply the concepts that are critical to the discipline should outperform those who are not applying the knowledge presented in the course. Two hypotheses were tested in this study: H1: The greater the number of concepts the students report employing, the greater the proportion of total, industry-wide profits the student team will achieve. 109 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 H2: The greater the number of concepts the students report employing, the greater the proportion of total, industry-wide revenues the student team will achieve. The same analyses were conducted to test the relationship between concepts used and revenue. Exhibit 2 shows the scatterplot for the two variables. RESULTS There was a considerable range in the performance of the nine simulation teams, both in terms of net income and total revenue. The Year 2 net income for the student teams (as a proportion of total industry profits) ranged from .017 to .239. That is, the net income of the most profitable team accounted for almost one-fourth of all the profits in the industry. Year 2 total revenues were far less dispersed, but still had quite a wide range – from .056 to .134. There was also a considerable range in the utilization of the marketing concepts presented in the course. Of the total of 23 marketing concepts presented to the students, the number of concepts student teams reported using ranged from five to nineteen. Simple linear regression analysis was run to test the hypotheses. Exhibit 1 shows a scatterplot of the relationship between number of concepts each student team reported using and that team’s proportion of the total industry-wide profits. 5 10 15 20 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 Number of marketing concepts used Y ea r 2 - Sa le s Relationship between 2nd year sales revenues and the number of marketing concepts used Exhibit 2. The results are quite similar to those for income. The adjusted coefficient of determination was 48.1% and the p- value was 0.023. Again we can conclude with confidence that there is a statistically significant relationship between the number of concepts a team reported using and their sales performance. Knowing the number of concepts a student team reported using enables us to explain almost half of all the variability in revenues earned. Thus, there is strong support for hypothesis 2 - The greater the number of concepts the students report employing, the greater the proportion of total, industry-wide revenues the student team will achieve. 2015105 0.25 0.20 0.15 0.10 0.05 0.00 Number of concepts used Y ea r 2 - In co m e Relationship between 2nd year net income and the number of marketing concepts used Exhibit 1. DISCUSSION AND CONCLUSIONS This study is concerned with what instructors can hope to accomplish through the use of a simulation. While researchers have identified many possible objectives for using a simulation (Hemmasi, M. and L.A. Graf, 1992; Keys, B. and J. Wolfe, 1990; Miles, W.G., Jr., et.al., 1986; Parasuraman, A., 1981), it seems likely that almost all instructors using games hope to stimulate learning at the application and analysis levels of Bloom’s hierarchy. By putting students into the position of managers we hope they will learn how to apply the concepts of the discipline. The adjusted coefficient of determination was 52.4% and the p-value was 0.017. Since the p-value is less than 0.05 we can conclude with confidence that there is a statistically significant relationship between the number of concepts used and the team’s income performance. The adjusted r2 of 52.4% tells us that knowledge of the number of concepts a student team reported using enables us to explain over half of the variability in a team’s income performance. Based on this sample information there appears to be a strong support for hypothesis 1 — the greater the number of concepts the students report employing, the greater the proportion of total, industry-wide profits the student team will achieve. It is not simply the application of the concepts that instructors seek, however. In addition, instructors would like the simulation to reward those students who do apply the concepts they learn in the classroom. If students to see the relationship between the application of concepts and performance, they will embrace the value those concepts bring to a business activity. This positive relationship will also reinforce the students’ efforts at analysis and application and provide them with motivation to go beyond simple knowledge and comprehension of course materials. 110 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 This study suggests that there is a relationship between the application of marketing principles and concepts, and consequent performance on the simulation. The results of the study demonstrate a strong relationship between the number of concepts students report using and the profit and sales performance of their company. A second limitation of the current study is that the sample size was quite small – there were a total of only nine companies. While the relationships found were very strong, a larger sample size would have been desirable. Yet another limitation is that the findings are based on a single simulation. It is entirely conceivable that some simulations would show strong, positive relationships while others – especially those employing models where the link between the application of a discipline’s principles and performance in the simulation is weak – would not show a relationship. We also would suggest that the strong relationship between the application of the concepts and profits reflects a deep comprehension of these concepts. If the relationship between the application of the concepts was limited to sales revenues, questions could be raised about whether students recognize the concepts are a means to an end (e.g., firm profitability and sustained existence), not an end in themselves (e.g., sales volume regardless of profitability). We contend that the significant relationship between the application of the concepts and profits indicates the students’ comprehensive analysis and focused application of the course concepts presented to them. The remedy for the latter two limitations is, of course, quite straightforward. Replication of this study by other simulation users and with other games is needed. REFERENCES Anderson, P.H., and L. Lawton (1997) “Demonstrating the Learning Effectiveness of Simulations Where We Are and Where We Need to Go.” Developments in Business Simulations and Experiential Exercises. Volume 24, 68-73. These findings provide a powerful validation for simulations. In answer to the question of whether applying the principles and concepts of a discipline results in positive results in a simulation, the answer appears to be an emphatic “yes”. The study demonstrates that simulations are a useful tool for operating at the application and analysis levels of Bloom’s hierarchy, levels at which traditional classroom lectures are thought to be weak. Whether simulations are better than other competing pedagogies such as cases remains an open question. Bloom, B.S., M.D. Englehart, E.D. Furst, W.H. Hill, and D.R. Krathwohl (1956), Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1: Cognitive Domain, New York: David McKay Company, Inc. Burns, A.C., J.W. Gentry, and J. Wolfe (1990), “A Cornucopia of Considerations in Evaluating the Effectiveness of Experiential Pedagogies.” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 253- 278. LIMITATIONS OF THE STUDY The principal limitation of this study is its reliance on the self-report of the students for the number of marketing concepts they used to guide their decision-making. The findings would be much stronger if we had some objective measure of the number of concepts the students actually used. This limitation is akin to one of the main criticisms leveled at the bulk of research on the efficacy of simulations – that the research relies on student perceptions rather than objective evidence. This study is a bit different, however. Most studies have used student perceptions as the dependent variable. A common question has been something along the lines of, “How much do you think you learned from participating in the simulation?” Questions of this type are fraught with problems. Regardless of how conscientious we attempt to be, providing an accurate subjective assessment of our own learning is a difficult task. In the current study, the question of whether students applied a particular marketing concept is much more concrete than how much was learned. Students’ opinions as to whether they exercised a behavior should be much more reliable than their opinion concerning the more ambiguous notion of how much they learned. It seems likely that students would be capable of providing a considerably more accurate judgment for a question of this type. Gentry, J. W. McCain, K. C., and A. C. Burns (1979), “Relating Teaching Methods with Educational Objectives in the Business Curriculum.” Developments in Business Simulations and Experiential Exercises, Volume 6, 196-198. Gosen, J., J. Washbush, & T. Scott (2000). “Initial Data on a Test Bank Assessing Total Enterprise Simulation Learning.” Developments in Business Simulations and Experiential Exercises, Volume 27, 166–171. Gosen, J. and J. Washbush (1999), “Perceptions of Learning in TE Simulations.” Developments in Business Simulations and Experiential Exercises, Volume 26, 170–175. Gosenpud, J. and J. Washbush (1994), “Simulation Performance and Learning Revisited.” Developments in Business Simulations and Experiential Exercises, Volume 21, 83-86. Gosenpud, J. and J. Washbush (1993), “The Relationship Between Total Enterprise Simulation Performance and Learning.” Developments in Business Simulations and Experiential Exercises, Volume 20, 141. Hemmasi, M. and L.A. Graf (1992) “Managerial Skills Acquisition: A Case for Using Business Policy 111 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Simulations.” Simulation & Gaming, Volume 24, Number 4, 298-410. Keys, B. (1976), “A Review of Learning Research in Business Gaming.” In B.H. Sord (ed.), Computer Simulations and Learning Theory, 173-184. Keys, B. and J. Wolfe (1990) “The Role of Management Games and Simulations in Education and Research.” Journal of Management, Volume 16, Number 1, 311. Miles, W.G., Jr., W.D. Biggs, and J.N. Schubert (1986) “Student Perceptions of Skill Acquisition Through Cases and a General Management Simulation: A Comparison.” Simulation and Games 17, 1: 7-24. Miller, H.E., P.L. Schumann, P.H. Anderson, & T.W. Scott (1998). Maximizing Learning Gains in Simulations: Lessons from the Training Literature. Developments in Business Simulation and Experiential Exercises, 25: 217–223. Schellenberger, R.E., J.A. Hill, and R.B. Keusch (1989), “An Exploratory Study of the Effect of Strategic Emphasis in Management Games on Attitudes, Interest, and Learning in the Business Policy Course.” Developments in Business Simulations and Experiential Exercises, 16, 178. Teach, R. and G. Govahi, (1988) “The Role of Experiential Learning and Simulation in Teaching Management Skills,” Developments in Business Simulation & Experiential Exercises, eds. P. Sanders and T. Pray 15: 65-71. Whiteley, T. R. and A. J. Faria, (1989) “A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation,” Developments in Business Simulation & Experiential Exercises, 16: 78- 82. Wolfe, J. (1990). The Evaluation of Computer-based Business Games: Methodology, Findings, and Future Needs. In J.W. Gentry (ed.), Guide to Business Gaming and Experiential Learning (pp. 279–300), New York: Nichols/GP Publishing. 112 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 113 Appendix A. How Did Your Company Determine Marketing Strategy? In this course we spoke of many things that should influence marketing strategy. Which of the following did you explicitly think about and actually use as a guide to your decision-making? Topic Yes No Selecting a target market (or markets) Organizing the 4 Ps around the target market Building a coherent marketing mix (making certain the elements of the marketing mix were internally consistent, a good blend) Attempting to develop a distinctive competitive advantage for your products Attempting to develop a product position for each of your products The alternative strategies for above average profits (e.g., Porter’s Model – cost leadership, focus, differentiation) Using a “matrix” (The Boston Consulting Group Matrix, The GE Business Screen) to guide marketing spending The implications of PIMS (large market share, “don’t get caught in the middle”, above average quality, etc.) Using military strategies (attacking and defending) Allocating your resources according to the profitability of the product Allocating your resources according to the profitability of the territory The need for marketing research to guide your decision-making Considering the cost vs. benefit of information Using financial statements to guide your decision-making Using the stage of the product life cycle to guide your marketing mix decisions Using the type of product (convenience, shopping, specialty) to guide your mix decisions The advantages and disadvantages of alternative promotional media Push vs. pull promotional strategy The effect of the elasticity of demand on pricing strategy Penetration vs. skimming pricing Did you actually ever calculate a break-even point? Did you attempt to use marginal analysis for pricing The need to integrate marketing decisions with the other activities of the firm (the need to see marketing as an integral part of the overall operations of the firm) Table of Contents Volume 29, 2001 Threshold Marketer A Family Of Marketing Simulations: Basic Marketer And Advanced Marketer Team Mode And Solo Mode Globalization As An Extended Experiential Exercise The Benefits And Planning Considerations Of Short Term Study Abroad Programs How 2 Setup Your Office Computer To Run Linux For Teaching E-Commerce Without Messing Everything Else Up Incorporating Cosmopolitan-Related Focus-Group Research Into Global Advertising Simulations Demonstration Of Advanced Features In Computer-Assisted Gaming Of International Business A Comparison Of Discrimination-Based Versus Conventional Simulation Game Scoring A Universal Mathematical Law Criterion For Algorithmic Validity The Impact Of Public Policy On Innovation: A Simulation Project For Research And Teaching Participant Identification Of Competitors In A Marketing Simulation Competition Simulation Research In The Hospitality Industry "Computer Simulation, Games And Roleplay: Drawing Lines Of Demarcation" Simulation Distribution Alternatives: Author/User Considerations Managing The Curiosity Gap Does Matter: What Do We Need To Do About It? Use Of External Interventions In A Computer Based Simulation Putting Service Learning Into Orbit It's A Wonderful Life: Simulating The Golden Years Adventures In Creating An Outdoor Leadership Challenge Course For An Emba Program Vbotz: A Pedagogical Cross-Disciplinary, Multi-Academic Level Manufacturing Corporate Simulation Use Of Computer Modeling In Management Accounting Is Simulation Performance Related To Application? An Exploratory Study Learning Cooperatively May Not Be Learning Collaborately! Perception Is Reality: Sharing Frames International Management Virtual Teamwork: A Simulation Financial Plan For Your Life And Career Goals Using Project-Based Experiential Learning Groups In The Principles Of Marketing Course Futures Course: Learning How To Anticipate The Future Of Business Interactive Online Strategic Market Planning With The Web-Based Boston Consulting Group (BCG) Matrix Graphics Package Strategy Learning In A Total Enterprise Simulation Investigation Of The Impact Of Decision Parameters For A Dutch Auction Simulation For Ipo Issues Integrating In-Class Learning With Out-Of-Classroom Experiences Through A Managerial Competency Development Framework War And Peace: Managing Students Learning Experience In A Competitive Simulation Game Virtually Experiential Classrooms Exercise: Conducting Role Plays Using Student Generated Cases Procedural Justice And Acceptance In Group Decision Making The E-Commerce Game: A Strategic Business Board Game Does Student Preparation Matter In A Simulation? A Comparison Of Pedagogical Styles The Game Of Business - A Weekend MBA Course Volume-Dependent Money Exchange Model For Gaming Simulations Implementing Service Learning For Accountants: The Not For Profit Project To Teach Vikings To Behave Among Mandarins: Lessons From Teaching With A Simulation Model Of Applied Business Ethics In International Management Maze Bright Teachers In The Classroom The Validity Investigation Of A Test Assessing Total Enterprise Simulation Learning What Makes Strategy Possible: An Illustration Using Paper And Scissors Learning Micro-OB Skills While Making Top Management Decisions In A Multinational Industrial Firm The Absel Research Heritage And The Bkl: Leveraging Their Value For Future Research New Product Development (Npd) Simulations: Some Challenging Questions And Tough Modeling Issues The Biofeedback Stress Test The Power Circle Exercise Total Enterprise Simulation Learning Compared To Traditional Learning In The Business Policy Course A Business Game Distance Education Application: Learning Outcomes And Experiences Is the Tobin's Q a good Indicator of a Company's Performance? A Critical Examination of the "Experiential" Premise Underlying BUsines Simulation Usage