ON THE VALUE OF BUGS IN SIMULATION ENVIRONMENTS Developments in Business Simulation and Experiential Learning, Volume 31, 2004 ON THE VALUE OF BUGS IN SIMULATION ENVIRONMENTS William D. Biggs Arcadia University biggs@arcadia.edu Annette L. Halpin Arcadia University halpin@arcadia.edu ABSTRACT This article looks at the issue of realism with respect to the use of simulations. The authors put forth the propositions that BUGS (Basic Unplanned Glitch Situations) in simulation environments are not necessarily detrimental to appropriate learning and that there are “real world” counterparts to BUGS. The article focuses on the sources of simulation BUGS and makes suggestions for using BUGS as learning tools. INTRODUCTION Almost from the introduction of the first practical business simulation in 1956, game designers and users have been admonished to make certain that simulations are bug free (i.e., that realism prevails). That there continues to be interest in realism and validity in simulation is evident from recent articles by Feinstein and Cannon (2002) who provide a table with 20 concepts related to simulation validation research and Dickinson (in press) who states: “It is an obviously essential property underlying the effectiveness of simulations that the outcomes of simulation game algorithms are valid.” For the most part the focus in earlier research on simulation realism has been on the validity of the relationships in simulation programs. A review of some of the stronger views, as found in The Bernie Keys Library CD, which contains the annual conference proceedings of the Association of Business Simulation and Experiential Learning from its founding in 1974 to the most recent conference (2003), clearly demonstrates this concern (Carvalho, 1992; Decker, LaBarre and Adler, 1987; Dickinson, Whitely, and Faria, 1990; Dittrich, 1976; Goosen, 1991; Norris (1986); Norris and Snyder (1981); Snyder, 1994; and Whitney, 1984;). Some writers raise questions about realism with respect to variables other than the validity of the model. Wolfe and Jackson (1989), for example, look at student perceptions as a factor related to realism. In addition, others (Frazer, 1980; Teach, 1990) question the need for realism in light of the objectives for the simulation. It is apparent from the literature, however, that there are many more writers who argue for realism than raise the question about whether there is even a need for realism. The major concern appears to be that bugs will detract from the learning experience. In this paper we argue that many game designers and users suffer from entomophobia, “a strong fear of, dislike of, or aversion to insects or bugs” (The Phobia Clinic, 2003). We do not dispute that realism is important and that care needs to be taken in designing and using simulations so that “correct” relationships and concepts are learned. We do believe, however, that they are not as detrimental as many writers assert. We agree with Teach (1990) that the objective is important but we go further by arguing that violations of realism, BUGS, are not necessarily bad since they can be used for learning purposes even though they were not planned. In addition we assert that in some cases, things perceived as BUGS may actually have a real world counterpart, or may not exist now but could in the future. In the next section we will provide some examples of BUGS and their sources. We will also discuss why, in some instances, BUGS may not really be BUGS as well as offer some suggestions as to how BUGS might be handled. BUGS IN SIMULATION ENVIRONMENTS We see BUGS in simulation environments as coming from three sources: (1) Designer/programmer actions; (2) Administrator/user actions; and (3) Student/player actions BUGS from designer/programmer actions result from two types of problems. The first problem is erroneously programmed theoretical relationships. The second problem is oversights in design that cause the simulation not to operate as intended. An example of an erroneously programmed theoretical relationship occurred in one simulation a number of years ago. This simulation was supposedly theoretically correct but it actually contained an upward sloping demand curve. The higher the price set, the more units sold. Given that such a relationship violates generally accepted economic theory, the simulation was found to be defective. But couldn’t this have been a useful learning experience? If students discovered this relationship and recognized that it was the opposite of the theory, wouldn’t this suggest that they learned the theory. Shouldn’t such analysis be rewarded? It is also the case that we could use this situation to distinguish between short run and long run situations when viewing possible events in simulations. There are, in fact, short- term situations, such as an inflationary environment in which consumers rush to purchase before prices go even higher, when an upward sloping demand curve might exist. 176 mailto:biggs@arcadia.edu mailto:halpin@arcadia.edu Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Finally, with respect to realism, we believe that we are sometimes in the same position as the King of Siam in the musical play, The King and I (Rodgers and Hammerstein, 1956). In the song, “A Puzzlement,” he states: "There are times I almost think I am not sure of what I absolutely know. … In my head are many facts of which I wish I was certain I was sure.” There was a period of time when economic theory held that it was not possible to have inflation and stagnation at the same time. During the 1960's, however, we in fact encountered both situations simultaneously and a new concept, stagflation, was born. If a simulation designer or administrator created such a situation prior to its actual occurrence in the real world, he or she would have been accused by students and colleagues of creating an unrealistic situation. In fact, the appearance of such BUGS in the early 1960’s might have prepared students for the future that was to occur in a few years. In other situations, conditions may change and create whole new business opportunities. We can look at changes in the airline and banking industries after deregulation as examples. In the airline industry when new regional/local airlines entered the market place, national airlines went to the hub and spoke system. In banking, eliminating regulation Q (a limit on interest rates paid to depositors) dramatically changed the nature of competition between banks and savings and loans. In another banking related area, changes in tax laws have spawned a host of new products for consumers. For example, the elimination of the tax write-off for interest expense other than mortgage interest resulted in the creation of home equity loans. There is no evidence that lawmakers or others were prepared for the creation of this substitute product when the tax laws changed yet today we have this new product in the market. In a simulation environment, the designer and/or administrator may create situations that create new possibilities/opportunities for firms that do the analysis of the conditions under which they are operating. All of these examples illustrate the point that conditions can change dramatically in the real world so why not in "simuland" (Wolfe and Jackson, 1989)? Simulation designers sometimes state the parameters for various decisions, yet fail to program the simulation accordingly. For example, in one simulation, production was to be limited by the lesser of raw materials available, plant capacity plus 20% (overworking the plant), or workforce plus 20% (using overtime). What was discovered was that if a production decision violated all three constraints then production levels greater than capacity plus 20% and/or workforce plus 20% could be achieved with the real limiting factor being the availability of raw materials at the plant. The game designers had assumed that no one would violate all three constraints simultaneously so when testing the program they only did paired comparisons of the restrictions that permitted excessive production decisions to be accepted by the program. One team of students discovered this bug and brought it to our attention. They also realized that even though they were charged at the overtime rate for the extra units produced, they were still in a good position because sales potential was higher than supply for the industry. Thus, their analysis not only detected the BUG but also involved assessing its importance. We agreed that they could produce at capacity plus 25% for the next quarter after which we corrected the program. In this instance we rewarded the team for their identification of the BUG and analysis of the situation. Interestingly, no other team detected this error. In this same simulation there was a failure in the program to check the number of salespeople a firm had when it fired salespeople so a firm could fire salespeople even when it had none. The salary costs for these ‘negative’ salespeople was negative and became a positive cash flow for the firm. Thus, the effect on the income statement was to lower total expenses, thereby increasing profits; the effect on the balance sheet was that it increased the firm’s cash balance; and the effect on cash flows was positive. This BUG was detected by one firm, which took advantage of it. When we become aware of such BUGS in a simulation package we take action to reward the students who bring them to our attention while trying to avoid penalizing those who may have been hurt by doing what the simulation manual instructed or was consistent with commonly accepted business theories. In some cases a team may detect a BUG that we are not able to correct, at least not in the short term. In such situations we take one of two actions depending upon the magnitude of the impact of the BUG. First, we may announce the problem to the class and say that the teams may not take advantage of the situation. We then check the decisions each quarter to make certain that no violations take place. Second, we may announce the situation to the class and let all teams make use of it. In both approaches we give the "discovery team" two quarters to take advantage of the BUGS before we go public. BUGS due to administrator actions come from two sources. First, administrators may have an inadequate understanding of how a particular relationship operates in the simulation and therefore construct an unplanned outcome. Second, administrators may create relationships between game parameters that have unintended outcomes We once created what we thought was an unrealistic situation due to our misunderstanding of how the Business Week Index (BWI) worked in the simulation, Tempomatic IV: A Management Simulation (Scott and Strickland, 1984). We knew that the index would lower industry sales potential if it were negative and increase sales potentials if it were positive. We misunderstood, however, how the magnitude of the BWI influenced sales potentials for the firms. Prior to starting the simulation we selected BWI figures for each of the simulation quarters. Upon examining the results for one quarter, we discovered that the BWI used for that period had caused total sales for the industry to be negative (i.e., customers were returning products more rapidly than new sales were being made). The simulation still ran perfectly but with negative sales, increases in inventory, etc. Since we had not yet returned the results to the students, we changed the BWI and did a rerun to eliminate this "unrealistic" and "impossible" situation. But was it really impossible that in the short run sales in an industry could be negative due to unusually high returns? In a conversation with the president of a company that made narrow- aisle forklifts (Raymond, 2001), this unrealistic situation was mentioned. He responded by saying it was not unrealistic since in his industry just such an event had occurred. He related that there had been a threat of an industry-wide strike and that this resulted in customers placing large orders in November and December to avoid not having their orders filled in January. The strike was settled, however, so during the month of January the 177 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 industry, as a whole, experienced more cancellations of orders than new orders placed. Thus, what we had assumed was an impossible real world situation did, in fact, have real world potential. It seems to us, therefore, that one could argue that almost any condition might occur in the real world and as long as the students have the information necessary to make forecasts and analyze a situation, one could let such "unrealistic" situations occur. The role of the administrator in these cases would be to offer examples of when a seemingly unlikely event might be possible. In one instance, we unintentionally structured the spread between the price received for a new stock issue and the price paid for retiring stock in such a way that it encouraged teams to engage in stock manipulation. Typically, firms try to manage costs related to plant expansion so that they don't result in large losses in a quarter since this will drive their stock price down. In the situation we created, one team discovered through analysis that by adding plant capacity in large rather than small increments they would incur large losses in a quarter, which caused their stock price to decline dramatically. In the next quarter with a lower stock price they repurchased a large number of shares. They would also make certain that they operated efficiently in the next quarter so they achieved a profit which when coupled with the fact that they had fewer shares outstanding meant their stock price was driven up. In the next quarter they proceeded to issue additional shares of stock, which brought cash into the firm to pay for the expansion. While one could argue that they had engaged in good management, the fact is that they were increasing their stock price more by manipulation than by operating the firm efficiently and effectively. At the time this occurred it was unusual for real world companies to engage in frequent repurchasing and issuing of their stock so it appeared this was an unrealistic situation. Never the less, it was decided that the situation could continue and after two more quarters we brought this practice to the attention of the other firms in the industry. Interestingly, most teams did not take advantage of this situation even after it was brought to their attention. This situation again represents a scenario in a simulation environment that was somewhat unrealistic at the time but which today is a more frequent real world phenomenon. BUGS from student actions also derive from two sources. First, due to a lack of knowledge or perceptions about the real world, students may erroneously expect to see a certain set of circumstances/parameters in the simulation environment. When their assumptions are mistaken they may conclude there is a glitch where one does not actually exist. Second, students may take actions themselves that create situations that administrators had not planned. As Wolfe and Jackson (1989) point out, students bring their own perceptions of reality to a gaming situation. It is possible, therefore, that something which is quite realistic might be perceived as unrealistic by the game player. We are reminded of a student who challenged Dick Cotter a number of years ago concerning the economic conditions he had created that year for the International Business Policy Game Competition. The student claimed that Dick had set interest rates at levels that could "never happen in the real world." Dick pointed out that there was a period of time in fairly recent history when the real world behaved exactly as the simulation had. In fact, he had used real world numbers from that exact time period. Thus, the simulation environment was realistic but due to a lack of student knowledge and/or false assumptions about the world the situation was perceived as unrealistic. If students make their perceptions known to the instructor, the misconceptions can be corrected and used as a basis for learning. The worst-case scenario here is that they do not discuss the topic with the instructor. They then continue to believe the simulation is unrealistic and that their erroneous perception of the real world is correct. As we noted at the outset, many writers argue quite strongly that students/players must perceive the simulation as realistic if they are to be engaged and willing to learn. But is it actually the case that students’ perceptions of realism really have this type of impact? Isn’t it possible that students see some aspect as unrealistic but are motivated to analyze the situation so that they can make effective decisions and perform well? Couldn’t such BUGS contribute to student analysis and learning? In addition, Wolfe and Jackson (1989) provide some evidence about student perceptions of realism that is disturbing, at least to us. They deliberately introduced a “glitch” into a simulation and found that: “the presence of the glitch had no discernible effect on the players’ perception of the game’s realism nor on each team’s economic performance” (Wolfe and Jackson, 1989:34). This result raises serious questions about students’ perceptions of realism and the degree to which realism is present in a game. Might we be better served if we encouraged students “to think outside the box” and do the analysis of the situation so that glitches are found? In some cases BUGS arise as the result of student actions during game play. During one semester we had a firm report to its board of directors that they had entered into a price fixing arrangement with two other firms in the industry. In fact, this was true, and while we thought it might be happening, we hadn’t been able to verify it. This situation, while unplanned, was interesting because it required the instructor to discuss a number of items with the class. First, the reactions of the board members had to be discussed. A few threatened to resign if the firms continued such behavior. While no one did resign the extent to which the board members were bothered by the student actions provided for good class discussion. Another offered to visit the managers in jail, which also sent the message that this behavior was not acceptable. Second, during a class soon after these revelations, the instructor pointed out three problems with price fixing as engaged in by these firms. First, price fixing is an illegal activity. Second, in a market-based economy there is an ethical dimension to the act of price fixing. Third, these firms did it badly. The fact is that none of the firms involved in the price fixing made a profit. They were concerned that if they set the price too high it would be noticed by the instructor so while they jointly set prices, they set them at a level that was not profitable. An interesting aside here is that this is very similar to what happened in the price fixing case involving Westinghouse and General Electric in the 1960’s. The firms were so concerned that they would get caught that they set prices at a level that still resulted in their incurring losses. A second example of an unplanned situation occurred when someone other than the instructor posted a notice on a bulletin 178 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 179 board outside the instructor’s office that stated demand would decline 50% in the next quarter. In the past, notices were posted on this bulletin board by the instructor and this particular message had an appearance virtually identical to those posted in the past. One team chose to believe the message, made its decisions, and lost significant sales and profits that period. When they learned the message was false, they were incensed and wanted an inquisition. While we had some sympathy for their position, the fact was that they had three data points that indicated demand was going up and they chose to believe the fourth data point that was an aberration. We also pointed out to them that in the real world firms sometimes put out misinformation. Needless to say they did not find our response satisfying. CONCLUSION We have argued in this article that BUGS in simulation environments, while unintended, may not necessarily be bad. There are some BUGS that may be beneficial because they contribute to the learning experience in a positive manner whether by design or by chance. BUGS have provided these authors opportunities to discuss with players topics in corporate strategy including economic theory (supply/demand relationship), operations management (setting production levels), industry indices (BWI), and equity financing (new stock issue). We have also addressed issues related to legal and ethical behavior such as stock price manipulation, price fixing, and issuing misinformation. We have offered comments on the various sources of BUGS and guidelines on how these situations can be handled. We hope that game designers and users will reassess their assumptions regarding the need for ‘realism’ in simulation environments. While a desirable and important element, designers unknowingly may be restricting the outcomes of their simulations by allowing only for those environments/relationships that have occurred in the real world. In any event it is likely that BUGS will continue to occur in simulations. How designers, instructors, and users manage these unusual events will likely influence the approval rating of this teaching tool. REFERENCES Note: All of the ABSEL Proceedings articles can be found on The Bernie Keys Library CD that is published annually. This CD contains all the articles from 1974 to date. The CD also contains the Guide to Business Gaming and Experiential Learning (1990), J.W. Gentry (ed.). Carvalho, G.F. (1992). A New market demand model for business simulators. Developments in Business Simulation & Experiential Exercises, J. Gosenpud and S. Gold (eds.), 19, 39-43. Decker, R., LaBarre, J. and Adler, T. (1987). The exponential logarithm function as an algorithm for business simulation. Developments in Business Simulation and Experiential Exercises, L. Kelly and P. Sanders (eds.), 14, 47-49 Dickinson, J.R. (in press). A mathematical law for assessing outcome values for games. Simulation & Gaming. Dickinson, J.R., Whitely, R.R., and Faria, A.J. (1990). An empirical investigation of the internal validity of a marketing simulation game. Developments in Business Simulations & Experiential Exercises, 17, 47-52. Dittrich, J. E. (1976). Realism in business games: A three game comparison. Computer Simulation and Learning Theory. Proceedings for the Association for Business Simulation and Experiential Learning, B. H. Sord (ed.), 3, 273-280. Feinstein, A. H. and Cannon, H. M. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33, 425-440. Frazer, J. R. (1980). Some issues in game design. Experiential Learning Enters the 80’s: Proceedings of the Association for Business Simulation and Experiential Learning, D. C. Brenenstuhl and W. D. Biggs (eds.), 7, 184-186. Goosen, K. R. (1991). Increasing simulation realism through the modeling of step costs. Developments in Business Simulation & Experiential Exercises, W. Wheatley and J. Gosenpud (eds.), 18, 38-43. Norris, D. R. (1986). External validity of business games. Simulation & Gaming, 17, 447-459. Norris, D. R. and Snyder, C. A. (1982). External validity of simulation games. Simulation & Gaming, 13, 78-85. Norris, D. R. and Snyder, C. A. (1981). External validation: An experimental approach to determining the worth of simulation games. Developments in Business Simulation & Experiential Exercises, W. D. Biggs and D. J. Fritzsche (eds.), 8, 247-250. Rodgers, R and Hammerstein, II, O. (1956). A Puzzlement. The King and I. 20th Century Fox Corporation). Scott, Jr., C. R. and Strickland, III, A. J. (1984). Tempomatic IV: A Management Simulation. Boston, MA: Houghton Mifflin Company. Snyder, S. J. (1994). An assessment framework for determining the effectiveness of total enterprise simulations. Developments in Business Simulation & Experiential Exercises, P. Thavikulwat and J. D. Overby (eds.), 21, 16- 19. Teach, R. D. (1990). Designing business simulations. Guide to Business Gaming and Experiential Learning. J. W. Gentry (ed.). London, England: Nichols/GP Publishing. The Phobia Clinic (2003). Fear of Bugs. http://www.changethatsrightno.com/problem_detail.asp?Ph obia ID=1531&SDID=819 Whitney, G. (1984). A comparison of two business strategy simulations for microcomputers. Developments in Business Simulation & Experiential Exercises, D. M. Currie and J. W. Gentry (eds.), 11, 258-260. Wolfe, J and Jackson, R (1989). An investigation of the need for algorithmic validity. Simulation and Gaming, 20, 272-291. http://www.changethatsrightno.com/problem_detail.asp?Phobia http://www.changethatsrightno.com/problem_detail.asp?Phobia Table of Contents Volume 31, 2004 Controlling the Complexity and Orenting Target Groups by a Modular, Server-Based Business Game System Learning Network Demonstration: Delivering Business Education in a Distance Learning Environment Economic Evolution, Human Capital Investment, and Adult Distributed Electronic Learning: A Literature Review Designing a Globalization Simulation to Teach Corporate Social Responsibility Developing and Teaching an Online / In-Class Hybrid: A Demonstration A Model for Evaluating Online Instruction An Evaluation of a Distributed Learning Course: A Students'-Eye Perspective Blended Learning Strategy Improved Business Writing Skills How to Receive and Process Attachemnts while Greatly Reducing the Risk of Viruses and Trojans Introducing Online Components to a Class: How to Increase teh Likelihood of Success Teaching Strategic Communications Online: Using Learning Outcomes to Develop a Case-Based Course Implementing Distance Approaches to Education: A Panel Discussion for ABSEL: Las Vegas, 2004 MANDI: Learning Management Through Field Sales Experience An International Capital budgeting Experiential Exercise A Primer To Combating Terrorism: Playing It Safe While On Overseas Assignment (An Experiential Exercise) Integrating The Business Curriculum With A Comprehensive Case Study: A Prototype The Case Brief: A Model For Case Analysis, Writing And Discussion Technology Infused Pedagogy And Delivery – A Sure Bet? A Proposal For Panel Discussion Absel Conference 2004 Research Strategy And The Bkl: Getting The Most From The Absel Archives Simple But Effective: Rediscovering The Class Discussion Needle And Thread: An Activity For Examining Various Management Behaviors A Customized Excel Data Analysis System For Use In Undergraduate Marketing Research Team Leader Selection - Does It Matter? The Power Of Perspective: Reframing Your Framing Skills For Innovative Instruction In Leadership And Influence Exercise: How Should Merit Raises Be Allocated? An Online Situation For Problem-Based Learning In A Junior-Level Management Course The Eden Alternative As A Roadway For Change: A Service Learning Quality Improvement Project Avoiding Catastrophe: The Role Of Individual Accountability In Team Effectiveness Omega Systems: A Change Management Exercise The Risks And Rewards Of Providing Students A Structured Cheating Opportunity Experimentation With Assessment Techniques: A Proposal For Panel Discussion Using A 2 - Page Case To Introduce Concepts Of Business Strategy Interactive Session The Integration Of Appreciative Inquiry And Experiential Learning For Peak Performance Appreciative Inquiry Case Story: New York City Leadership Challenge Individual Achievement Versus Team Performance: An Empirical Study With Business Games Some Strategists Don't Learn Or Can't Learn Computer Simulation: A Design Architectonic On The Value Of Bugs In Simulation Environments Online Sales Forecasting With The Multiple Regression Analysis Data Matrices Package Simulation Exercises And Problem Based Learning: Is There A Fit? A Study Of Business Game Stock Price Algorithms Assessing Individual Performance In A Total Enterprise Simulation Information Use In A Business Game Determining The Value Of A Firm Unsorting Algorithms For An Ordered List And Its Application To Business Simulations Teaching Public Finance Management Through Simulation Antecedents Of Game Performance Student Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Implementation And Impacts Of The Balanced Scorecard: An Experiment With Business Games Impact: Shocking The Legacy Mindset Implementation Of The Eepad Framework Of Business Processes In An Accounting Information Systems Course Are Business Games Really Delivering What Students Are Led To Believe?? Reporting Lessons Learned: What Gets Reported; Who Gains Value Teacher Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Student Reactions To The Use Of A Computer-Based Simulation As An Integrating Mechanism For A Mba Curriculum A Cognitive Investigation Of The Internal Validity Of A Management Strategy Simulation Game The Casino Challenge: Making Simulation Delivery A Safe Bet! Accounting For Company Reputation: Variations On The Gold Standard Foreign Currency Hedging: A Simulation The Influence Of Variables Easily Controlled By The Instructor/Administrator On Simulation Outcomes: In Particular, The Variable, Reflection. Absel Awareness Among Business School Faculty Validating Business Simulations: Does High Market Share Lead To High Profitability? Simulation Debriefing Procedures Coaching And Business Simulations: A Formula For Success? A Seminal Inventory Of Basic Research Using Business Simulation Games The Influence Of Scorecard Evaluation On Decisions And Outcomes