A RANDOM-STRATEGY CRITERION FOR VALIDITY OF SIMULATION GAME PARTICIPATION Developments In Business Simulation & Experiential Exercises, Volume 21, 1994 35 A RANDOM-STRATEGY CRITERION FOR VALIDITY OF SIMULATION GAME PARTICIPATION John R. Dickinson, University of Windsor A. J. Faria, University of Windsor INTRODUCTION It has now been over 35 years since the first business simulation game was used in a college class at the University of Washington in 1957 (Watson 1981). Since that time, the number and variety of business games available for classroom use has grown enormously. Interest among academics in the teaching and learning possibilities of business games has clearly grown as well. Presently, over 200 business games are being used by nearly 9,000 teachers at over 1,700 colleges offering business programs (Faria 1989). Empirical research in the area has been extensive. Comprehensive reviews can be found in Greenlaw and Wyman (1973), Keys (1976), Wolfe (1985), Miles, Biggs and Shubert (1986), and Randel, Morris, Wetzel and Whitehill (1992). Despite the widespread use of business simulations, an important and vexing issue regarding business games is whether or not participation is a meaningful experience. This paper introduces an original criterion for the validity of simulation participation that is akin to the random sampling error basis of statistical hypothesis testing. Specifically, whether the results of participants presumably acting on logical, analytical, thoughtful bases are significantly better than results obtainable on a random decision-making basis. If participants’ results are better than those obtained on a random basis, then this would suggest their strategy formulations are purposeful and systematic; that is, they are meaningful in this sense and, too, valid in this same sense. In this paper the random strategy criterion and its operationalization are first developed. The criterion is then applied to a substantial sample of simulation game participants. PAST RESEARCH Meaningfulness, as applied to business simulations, has taken on a number of specific interpretations as reflected in past research including: (1) the learning, or skills training, aspects of simulation gaming, (2) the relative merit of simulation games versus other teaching methods, (3) the external validity of business simulations, and (4) the internal validity of simulation game participation. Research into the learning or skills training aspects of simulation gaming dates back a number of years. The reported types of learning brought about through simulation participation include goal setting and information processing (Philippatos and Moscato 1 969; Greenlaw and Biggs 1974; Biggs 1975; Biggs and Greenlaw 1976), organizational behavior and personal interaction (Cangelosi and Dill 1965; Chisholm 1979), sales forecasting (Edwards 1987; Hall 1987; Neuhauser 1976; Snow 1 976), financial analysis skills (Faria and Nulsen 1976; Hall 1987), basic economic concepts (Edwards 1987), and selected quantitative skills (Whiteley and Faria 1989). The relative merit of simulation games versus other teaching approaches has been investigated by a number of researchers (Greenlaw and Wyman 1973; Keys 1976; Snow 1976; Waggener 1979; Wolfe 1985; Miles, Biggs and Shubert 1986; Hall 1987; Specht and Sandlin 1991; Washbush and Gosenpud 1991; Randel, Morris, Wetzel and Whitehill 1 992). Across all of the reported studies, simulation games were found to be more effective than conventional instructional methods in 75 of the comparisons, conventional methods of instruction were found to be superior in 27 of the comparisons, while no differences were reported in 58 of the comparisons. The reported studies included business as well as social science simulations. The external validity of a simulation model has generally been viewed as a measure of how well the model matches its real-world counterpart (Carvalho 1991; Mehrez, Reichel and Olami 1987; Parasuraman 1986; Stanislaw 1986; Watson 1981; Wolfe and Roberts 1986). In the classroom setting, two approaches have been used to investigate external validity. The first approach has focused on the correlation between a business executive’s simulation performance and his/her real-world performance. If the simulation is externally valid, a successful business executive should also be successful when participating in a simulation competition. A number of studies have generally supported the external validity of the simulations being examined (Babb, Leslie and Van Slyke 1966; McKinney and Dill 1 966; Vance and Gray 1967; Wolfe and Roberts 1 986). The second approach to measuring external validity uses a longitudinal research design. In this approach, a student’s business game performance is compared to some measure of subsequent business career success. Using this approach, Norris and Snyder (1982) did not find a significant correlation between business game performance and career success although two more recent studies have found such a correlation (Wolfe and Roberts 1986; Wolfe and Roberts 1993). The internal validity of simulation games has also been examined in two ways. The first approach basically states that if a simulation exercise is to be internally valid, better students should outperform poorer students. Several studies have supported this view of the internal validity of simulations (Gray 1972; Vance and Gray 1972; Wolfe 1987). A second, and more reasonable view of internal validity, examines whether participant decisions in a simulation competition, over time, conform to the environment of the simulation. While the dynamics of the simulation and the actions of competing companies will certainly influence participants’ decisions, the simulated environment too must be considered and, ceteris paribus, participants’ decisions should reflect or adapt to this environment. If this type of adaptive decision making does occur, the simulation can be said to be internally valid. Several studies of this nature (Faria, Dickinson and Whiteley 1 992; Whiteley, Faria and Dickinson 1991; Dickinson, Faria and Whiteley 1990) have been mildly supportive of the internal validity of the simulations examined. While each of the four approaches used in the past has merit, a new approach to examining simulation validity and meaningfulness is presented. The authors feel that the random-strategy criterion proposed is conceptually and logically more appealing. RANDOM-STRATEGY CRITERION In one major format of simulation gaming, participants are grouped into companies and companies are, in turn, grouped into industries. Companies within a given industry compete against each other with simulation participants managing the companies. The basic premise of the random- strategy criterion is that performance resulting from the presumably systematic strategy planning of participants should better performance resulting from random strategy “planning.” More specifically, if a given industry comprises both “real” companies as managed by participants and “random” companies whose strategies are determined randomly, the performance of the former Developments In Business Simulation & Experiential Exercises, Volume 21, 1994 36 should be better than the performance of the latter. Participants making planned, informed, systematic strategy decisions should achieve better company performance than achieved by a strategy made up of random decisions. In many instances of hypothesis testing, the fundamental criterion is the probability that observed data are attributable to chance in the form of random sampling error. It that probability is low, then the alternative general attribution is that the observed data reflect some systematic phenomena. In parallel fashion, if the performance of companies managed by simulation participants is not significantly superior to the performance of companies managed at random, this would suggest that the participants’ strategies do not derive from systematic, i.e., meaningful, decision making and the validity of the participants’ experience must be questioned. On the other hand, if real company performance is significantly better than random company performance, this would suggest that participant decision making is purposeful and meaningful. Logical Basis of the Criterion The expectation of superior performance of simulation participants’ companies is supported by at least four types of factors: internal consistency, targeted strategies, experience, and planning. First, an effective strategy would have a degree of internal consistency. In a marketing simulation, for instance, a company paying a high salary to salespeople would tend to pay a low commission. High expenditures on advertising or other elements of strategy should allow a higher price to be charged. A company charging a relatively high price should not choose a low price message in its promotion. And so on. Random decisions might well result in, say, a high salary for the sales force. But since this random decision is independent of the commission decision, the random commission decision could be either high or low, regardless of the salary level. The two random strategy components may or may not be internally consistent, while participants’ decisions would be expected to be consistent and, thus, more effective. Random high expenditures on advertising would not impact the random price level, the latter thus not necessarily being consistent with the former. And a low price message might be randomly selected regardless of the level of the random price. Targeted and otherwise directed strategies may take many forms. For instance, a real company may plan to gain a preemptive foothold or dominant share of a certain market segment. A real company may follow or counter strategies of a specific competing company. A real company may drastically reduce inventory of a specific product. A real company may intentionally devise strategies to mislead competitors. Random strategies would have no such specific targeting or directing capacity. A third factor in support of the effectiveness of participants’ strategies is the experience gained by participants. Participants may be able to discern trends in competitors’ strategies and, also, participants accumulate learning as to which strategies are more and less effective. Random strategies do not reflect trends or experience. And, fourth, participants may plan for the future. One straightforward instance of this factor is where the effectiveness of certain strategy elements is time related. For example, newly hired salespeople may not actually engage in selling activities during the period in which they are hired. They may undergo training in that period and commence selling in the next period. Promotion effects may accumulate and/or decay over time. Generally, a strategy that changes drastically from one period to the next may flat be as effective as a strategy that evolves more gradually. These types of effects are founded in the simulation environment itself and all of the effects are, in fact, part of the simulation used in the study described below. Random strategies do not reflect planning for the future. The random-strategy criterion is a conceptual development. Its merit is based on the logic underlying it. It simulation participation is not a meaningful experience, strategies formulated should tend to be unsystematic generally. More specifically, in the framework described above, strategies should tend to be unsystematic in the respects of lowered internal consistency, an absence of targeting, lessened reflection of experience, and lessened reflection of planning for the future. Resulting performance, then, should approach that of performance based on random strategies. To the extent that strategy formulation is systematic, i.e., purposeful and meaningful, performance should significantly exceed that of performance based on random strategies. An application of the criterion is described below, more for purposes of illustration than for the purpose of testing the criterion. HYPOTHESIS Based on the reasoning above, the hypothesis tested in applying the random-strategy criterion is that real companies will outperform random companies, both on a period by period basis and on a cumulative basis. Hai: In Period i the performance of companies managed by participants will be greater than the performance of companies using random strategies. Hb: The cumulative performance of companies managed by participants will be greater than the cumulative performance of companies using random strategies. DATA COLLECTION The random-strategy criterion was applied using LAPTOP: A Marketing Simulation (Faria and Dickinson 1987). Participants in the LAPTOP simulation take on roles of the top marketing managers for a medium size company that manufactures laptop computers. Participants are responsible for their company’s entire marketing program. This includes the product, price, place, and promotion decisions for their companies. Companies may sell one or both of two models of laptop computers in one or two geographic territories. Together these comprise four product-market segments. All told, 32 strategy decisions affect demand. In addition, 1 2 separate market research studies maybe requested. LAPTOP proceeds on a period by period basis. Participants formulate their strategies and receive the results of those strategies in the form of updated financial statements, market share data, and other types of information. With those results and considering the new status of their company, participants again formulate their strategies. Each period companies realize after tax earnings. In LAPTOP these earnings are transformed to earnings per share which is the bottom line measure of performance for a given period. These period earnings per share may also be accumulated as a competition progresses, yielding cumulative earnings per share (CEPS). It is CEPS that is the usual final measure of overall company performance at the end of the competition. Operationalizing Random Decisions In the usual fashion, students involved in this study were grouped into companies and companies were grouped into industries. For Developments In Business Simulation & Experiential Exercises, Volume 21, 1994 37 purposes of this study, an additional researcher-controlled company was added to each industry. This was the random strategy company. The range or available categories for each of the 32 decisions in LAPTOP is restricted. However, the allowable ranges are extremely wide. For instance, price may range from $0 through $9,999, expenditures on broadcast advertising may range from $0 through $999,000, sales force commission may range from 0.00% through 99.9%, and so on. Despite their being allowed, decision values over much of the range are clearly unrealistic. For instance, prices above about $3,000 (about four to seven time’s unit cost) result in zero unit sales. Due to diminishing marginal response, broadcast advertising expenditures above about $80,000 have almost no impact on unit sales. And commission percentages above about 20% increase expenses more than revenues. For this reason, then, random decision values could not be generated throughout the entire allowable range. In addition, considerable experience shows that during the competition different industries take on different strategy “mentalities.” Specifically, companies in some industries come to spend much more than companies in other industries. (In LAPTOP total market demand expands and contracts partly as a function of overall spending. Thus, profits may be realized whether or not companies in a given industry take on a spending mentality.) For this reason, too, a single set of random decision values could not be generated for researcher-controlled companies in all industries. A random broadcast advertising expenditure of, say, $40,000 might be in the ballpark of real companies’ expenditures in a growing, i.e., spending, industry, but much higher than real companies’ expenditures in an industry that has not grown appreciably. In order for random decision values to be within a range relevant to real competing companies within an industry, the following approach was taken to generate random decision values. For each decision, the highest and lowest values for the real competing companies defined a range. A random decision value was generated with a uniform probability across that range. Thus, if the highest and lowest prices were $2,000 and $1,200, respectively, a random decision value was generated between these two values. In this way, random decision values were kept within a range relevant to the real competing companies. Advertising message types and sales promotion types both comprise categories from which participants may select options. For these decisions, a type of message or promotion was selected at random. Four types of factors that should lead to real companies’ performances bettering those of random companies were described above. One of these was experience, whereby participants might identify trends or discern the sensitivity of the market environment to specific strategy variables. With the operationalization of the random strategies used in this study, the experience factor effect is mitigated somewhat. Random decisions to some degree tend to follow the collective wisdom of the real companies with respect to experience. However, the random decisions are not a function of the mean strategies of the real companies. The high-low range of which the random decisions are a function is not necessarily a good indicator of the collective wisdom of eli the real companies. The high-low range, obviously, reflects the strategies of only two real companies, not all of the companies comprising an industry. And these two real companies, just as obviously, represent opposing strategies, not consensual strategies. Dependent Variables As explained earlier, in LAPTOP earnings per share (EPS) is the overall measure of company performance for a given period. Cumulative earnings per share (CEPS) at the end of the competition is the final overall measure of performance. In this study, each period of the competition provides an opportunity to compare the performances of real and random companies and EPS is used for this purpose. In addition, CEPS at the end of the competition is used to reflect the total performance of companies over the entire competition. Sample Data were collected from four sections of an undergraduate introductory marketing course. All sections were taught by the same instructor using a common syllabus and format. The total of 660 students were grouped into 165 companies, each company managed by four students. Students selected their own group members. LAPTOP performance contributed 20% to the final course grade with performance being determined by individual period EPS and final CEPS. The 1 65 companies were grouped into 55 industries of three companies each. To each industry was added a fourth company, the researcher- controlled company using random strategies. As is a common practice, an initial trial period allowed participants a single opportunity to familiarize themselves with the general effectiveness of their strategies. After the trial period, results were discarded and the simulation started again. The actual competition extended over eight periods. In sum, the data comprised EPS for 1 65 real companies and 55 random companies for each of eight periods plus CEPS for each of the companies at the end of the competition. ANALYSIS For each period, average EPS was calculated across the 1 65 real companies and across the 55 random companies. As well, average CEPS for the real and random companies, respectively, was calculated at the end of the competition. The initial statistical hypothesis tested for Hai is that the vector of eight mean EPSs for the real companies is equal to the vector of eight mean EPSs for the random companies. The single independent variable is whether a company was managed by real participants or used a random strategy. “Subjects” in this study were the real and random companies. EPSs were measured for the same real companies, i.e., the same simulation participants, for each of the eight simulation periods. While it might appear that the random strategy companies were not the same companies from period to period, this is not the case. Though their strategies were random, each such company did indeed carry over its status, e.g., balance sheet, inventory levels, lagged and cumulative market response effects, sales force morale, etc., from period to period. The appropriate analysis for Hai, then, is a repeated measures or within-subject multivariate analysis of variance (Bray and Maxwell 1985, p. 69). RESULTS Mean EPSs for real and random companies are summarized in Table 1. The basic MA NOVA significance level was 0.063. Thus, it may be concluded that over all eight periods, companies managed by participants did indeed realize earnings different from (actually, greater than) earnings realized by random companies. The basic premise that the simulation participation experience is a systematic, Developments In Business Simulation & Experiential Exercises, Volume 21, 1994 38 i.e. meaningful one is supported. Mean EPS for the real companies exceeds mean EPS for random companies for each of Periods 2 through 8 TABLE 1 RESULTS SUMMARY Real Random Real Random Real/Random Period EPS $ EPS $ Difference p-value EPS Percent 1 -0.62 -0.52 -0.10 599 … 2 0.57 -0.20 0.77 0.84 … 3 1.16 0.35 0.81 .133 177 4 2.14 0.95 1.19 .053 225 5 3.11 1.58 1.53 .035 197 6 5.06 3.52 1.54 .051 144 7 6.52 4.68 1.84 .009 139 8 7.77 5.96 1.81 .027 130 Cumulative 25.70 16.31 9.39 .011 158 • one-tail t-test Utilizing directional univariate t-tests, mean EPS for the real companies was significantly (p<.10) greater than that for random companies in six of the eight decision periods. One of the exceptions was the first period, when participants likely engage in some exploration, and the other exception was nearly significant (p = .1 3) in Period 3. Starting with Period 3, real company mean EPS was at least 30 percent greater than random company mean EPS for each period. Thus, Hai is supported. Discussed earlier were several factors supporting the expectation of superior performance by simulation participants over random companies. Several of these involve learning by participants. As would be expected, then, real company mean EPS increased as the competition progressed and learning took place. Random company mean EPS also increased, no doubt reflecting the operationalization of random decisions based on the range of real companies’ decisions. However, as participants’ learning increased, their performance differential over that of random companies increased fairly regularly over the eight periods. The mean CEPS of real companies at the end of the competition was significantly (p<.02) greater than the mean CEPS of random companies. Overall, real companies earned an average of 58 percent more than did random companies by the end of the competition. Hb is supported. Additional Descriptive Results Within each industry three real companies competed against a single random company (as well as amongst themselves). Within each industry each period, then, there were 3 real versus random company competitions. Across the 55 industries and 8 periods, 1,320 real versus random company competitions took place. By chance alone, a real company’s EPS would exceed a random company’s EPS on one-half of these occasions. In fact, however, real companies’ EPSs exceeded random companies’ EPSs on 61 .8 percent of the occasions. For Periods 3 through 8, i.e., discounting the two initial periods when participants would be in the steepest part of their learning curve (along with the initial trial period), the percent of the 990 occasions when real companies’ EPSs exceeded random companies’ EPSs increased to 64.4 percent. DISCUSSION AND CONCLUSION Based on their superior performances over random strategies, it is clear in this study that participants’ strategies were systematic. i.e., purposeful and meaningful. Participant performance was not significantly superior during the first few periods of the competition, likely reflecting the unsystematic or exploratory character of early strategies. However, the trend toward superiority as the competition progressed is unmistakable. For the last five of the eight periods, real companies’ mean EPSs were significantly greater than the mean EPSs of random companies. Over the course of the entire competition, real companies on average earned 58 percent more than random companies. In head to head competition, real companies bested random companies between 62 and 64 percent of the time. In sum, it appears that the simulation participation experience was meaningful. The primary contribution of this paper is the conceptual development of a random-strategy criterion for the validity of simulation participation; validity in the sense that strategy formulation takes place on a systematic basis. Application of the criterion was illustrated using a popular marketing simulation and based on a substantial number of participants. REFERENCES Anderson, P. H. and L. Lawton (1990) “The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise”, in Developments in Business clear in this Experiential Learning, John Wingender and Walt Wheatley (eds.), Association for Business Simulation and Experiential Learning, 1 7, 6-10. Babb, E. M., M. A. Leslie and M.D. Van Slyke (1966) “The Potential of Business Gaming Methods in Research,” Journal of Business, 39 (Winter) 465-472. Biggs, W. D. (1975) “Some Impacts of Varying Amounts of Information on Frustration and Attitudes in a Finance Game,” in Developments in Business Simulation and Experiential Learning, R. H. Buskirk (ed.), 2, 23-30. Biggs, W. D. and P. 5 Greenlaw (1976) “The Role of Information in a Functional Business Game,” Simulation & Games, 7: 53-64. Bray, James H. and Maxwell, Scott E. (1985), Multivariate Analysis of Variance (Beverly Hills: Sage Publications). Butler, R. J. and A. Parasuraman (1977) “Degree of Uniformity in Achievement Motivation Levels of Team Members,” in Developments in Business Simulation and Experiential Learning, C. E. Neilsen (ed.), Association for Business Simulation and Experiential Learning, 1 63-1 68. Cangelosi, V. E. and W. R. Dill (1965) “Organizational Learning: Observations Toward a Theory,” Administrative Science Quarterly, 10 (2), 175-203. Carvalho, G. F. (1991) “Evaluating Computerized Business Simulators for Objective Learning Validity,” Simulation & Gaming, 22: 328-347. Chisholm, T. A. (1979) “An Examination of the Perceived Effectiveness of Computer Simulations in a Classroom Setting as Affected by Game, Environmental and Respondent Characteristics,” in Developments in Business Simulation and Experiential Learning, S. C. Certo and D. C. Brenenstuhl (eds.), 6, 103-109. Dickinson, J. R., A. J. Faria and T. R. Whiteley (1990) “The Impact of Market Structure, Versus Competitor Strategy, on Simulation Outcomes,” in Developments in Marketing Science, B. J. Dunlap (ed.), Academy of Marketing Science, 279-284. Edwards W. F. (1987) “Learning Macroeconomic Theory and Policy Analysis via Microcomputer Simulation,” in Developments in Business Simulation and Experiential Learning, Lane Kelley and Pat Developments In Business Simulation & Experiential Exercises, Volume 21, 1994 39 Sanders (eds.) Association for Business Simulation and Experiential Learning, 50-53 Faria, A. J., Dickinson, John R., and Whiteley, T. R. (1992), “A Measure of the Internal Validity of a Marketing Simulation Game,” Simulation/Games for Learning, Vol. 22 (December), 35 1-66. Faria, A. J. (1987) “A Test of Student Performance and Attitudes under Varying Game Conditions,” in Developments in Business Simulation and Experiential Learning Al Burns and Lane Kelley (eds.), Association for Business Simulation and Experiential Learning, 70-75 Faria, A. J. (1987) “A Survey of the Use of Business Games in Academia and Business,” Simulation & Games 18 (2), 207-225. Faria, A. J. and J. R. Dickinson (1987) LAPTOP: A Marketing Simulation. PIano, TX: Business Publications, Inc. Faria, A. J. and R. 0 Nulsen (1976) “Student Reactions and Evaluations Towards Differing Assignment Levels in a Marketing Simulation Competition,” in Developments in Business Simulation and Experiential Learning, B. H. Sord (ed.), 3, 26-35. Gosenpud, J. (1 989) “The Prediction of Simulation Performance as It Is Affected by Time,” Simulation & Games, 20: 3 19-350. Gray, C. E. (1972) “Performance as a Criterion Variable in Measuring Business Gaming Success,” Proceedings of the Southeastern Conference of the American Institute for the Decision Sciences, T S. Macklin (ed.), 146-151. Greenlaw, P. 5 and W. D. Biggs (1974) “Finansim: From Genesis to Revelations,” Proceedings of the Thirteenth Annual Symposium of the National Gaming Council, Ralph Day (ed.), 445-449. Greenlaw, P. 5 and F. P. Wyman (1973) “The Teaching Effectiveness of Games in Collegiate Business Courses,” Simulation & Games, 4 (2), 259-294. Hall, D. R. (1987) “Developing Various Student Learning Abilities via Writing, the Stock Market Game, and Modified Marketplace Game,” in Developments in Business Simulation and Experiential Learning, Lane Kelley and Pat Sanders (eds.), Association for Business Simulation and Experiential Learning, 84-87. Keys, B. (1976) “A Review of Learning Research in Business Gaming,” in Developments in Business Simulation and Experiential Learning. C. E. Neilsen (ed.), Association for Business Simulation and Experiential Learning, 76-83. McKinney, J. L. and W. R. Dill (1966) “Influences on Learning in 5 Simulation Game,” American Behavioral Scientist, 10 (Summer), 28- 32. Mehrez, A., A. Reichel and R. Olami (1987) “The Business Game Versus Reality,” Simulation & Games, 1 8 (3), 488-500. Miles, W. G., W. D. Biggs and J. N. Schubert (1986) “Student Perceptions of Skill Acquisition Through Cases and a General Management Simulation,” Simulation & Games, 10 (1), 75-86. Morrison, Donald F. (1967), Multivariate Statistical Methods (New York: McGraw-Hill Book Company). Neuhauser, J. J. (1976) “Business Games Have Failed,” Academy of Management Review, 1: 124-129. Norris, D. R. (1985) “Management Gaming: A Longitudinal Analysis of Two Decades of Use in Collegiate Schools of Business,” Proceedings of the 45th Annual Meetings of the Academy of Management, C. Furillo (ed.), 253-259. Norris, D. R. and A. Snyder (1982) “External Validation: An Experimental Approach to Determining the Worth of Simulation Games,” in Developments in Business Simulation and Experiential Learning, W. D. Biggs and Lee Graf (eds.), Association for Business Simulation and Experiential Learning, 247-250. Parasuraman, A. (1986) Marketing Research, Reading, MA: Addison- Wesley Publishing Company. Philippatos, G. C. and D. R. Moscato (1969) “An Empirical Study of the Learning Aspects of Experimental Game Playing,” AIIE Transactions, 1: 343-3 48. Randel, Josephine, B. A. Morris, C. D. Wetzel and B. V. Whitehill (1992) “The Effectiveness of Games for Educational Purposes: A Review of Recent Research,” Simulation & Gaming, 23: 261-276 Snow, C. C. (1976) “A Comment on Business Policy Teaching Research,” Academy of Management Review, 1: 133-135. Specht, L. B. and P. K. Sandlin (1991) “The Differential Effects of Experiential Learning Activities and Traditional Lecture Classes in Accounting,” Simulation & Gaming, 22: 1 96-21 1. Stanislaw, H. (1986) “Tests of Computer Simulation Validity,” Simulation & Games, 17: 173-192. Vance, S. C. and C. F. Gray (1967) “Use of a Performance Evaluation Model for Research in Business Gaming,” Academy of Management Journal 10: 27-37. Waggener, H. A. (1979) “Simulation vs. Case vs. Text: An Analysis of Student Opinion,” in Developments in Business Simulation and experiential Exercises, Bernard Keys (ed.), 1, 11 3-1 18. Washbush, J. B. and J. J. Gosenpud (1991) “Student Attitudes About Policy Course Simulations,” in Developments in Business Simulation and Experiential Learning, Walt Wheatley and J Gosenpud (eds.), 18, 105- 110. Watson, H. J. (1981) Computer Simulation in Business New York: John Wiley & Sons Wellington, W. E. and A. J. Faria (1991) “An Investigation of the Relationship between Simulation Play, Performance Level and Recency of Play on Exam Scores,” in Developments in Business Simulation and Experiential Exercises, Walt Wheatley and Jerry Gosenpud (eds.), Association for Business Simulation and Experiential Learning, 111- 116. Wellington, W. E. and A. J. Faria (1990) “The Effect of Decision Format and Evaluation on Simulation Performance, Decision Time and Team Cohesion,” in Developments in Business Simulation and Experiential Exercises, John Wingender end Walt Wheatley (eds.), Association for Business Simulation and Experiential Learning, 1 70- 175. Whiteley, T. R. and A. J. Faria (1989) “A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation,” in Developments in Business Simulation and Experiential Exercises, Tom Pray and John Wingender (eds.), Association for Business Simulation and Experiential Learning 78- 83 Whiteley, T. R., A. J. Faria and J. R. Dickinson (1991) “An Empirical Investigation of the Adaptive Behavior of Single-Player Teams to Simulation Game Parameters,” in Proceedings of the International Simulation and Gaming Association, 103-109. Wolfe, J. (1976) “The Effects and Effectiveness of Simulation in Business Policy Teaching Applications,” Academy of Management Review, 1: 47 56. Wolfe, J. (1978) “The Effects of Game Complexity on the Acquisition of Business Policy Knowledge,” Decision Sciences, 9: 143-155. Wolfe, J. (1985) “The Teaching Effectiveness of Games in Collegiate Business Courses: a 1973-1983 Update,” Simulation & Games, 16 (2), 251 288. Wolfe, J. and C. R. Roberts (1986) “The External Validity of a Business Management Game,” Simulation & Games, 17 (3), 45-59. Wolfe, J. and C. R. Roberts (1993) “A Further Study of the External Validity of Business Games: five-year Peer Group Indicators,” Simulation & Gaming, 24:21-34. Wolfe, J., D. D. Bowan and C. R. Roberts (1989) “Team-Building Effects on Company Performance: A Game Based Study,” Simulation & Games, 20: 388-408. Table of Contents Volume 21, 1994 ABSEL: The Way We Were and Need to Be The Intellectual Structure of ABSEL: A Bibliometeic Study of Author Cocitatons Over Time Activity-Driven Time in Computerized Gaming Simulations An Assessment Framework for Determing the Effectiveness of Total Enterprise Simulations Attributes of Learning Organizations: Simulating the Relationships Business Policy/Strategy Case Extension using Pro-Forma Planning: A Computer Based Model Complexity: Is it really that Simple? A Random-Strategy Criterion for Validity of Simulation Game Participation Enhancing a Computer Simulation with a Structured Reporting Environment Experiencing a Foreign Culture: A Cross-Cultural Simulation Group Cognitive Style and Computerphobia in Functional Business Simulations Human Issues in Technology Implementation Management Simulator Incorporating Advertising Strategy into Computer-Based Business Simulations: A Validation Study Increasing the Effectiveness of Performance Evaluation Through the Design and Development of Realistic Finance Algorithms The Packer-Feeder Game: A Commodity Market Simulator Relationships Between R&D and Profitability: An Exploratory Comparison of Two Business Simulations with Two Real-World, Technology Intensive Industries Simulation of the Predictive Value of Mammography Simulation Performance and Learning Revisited Strange Bedfellows: Competency Models and ACBSP Accreditation Standards Using a Business Simulation to Study the Determinants of Ethical Behavior What Simulation Users Think Players Should be Learning from the Simulations ADA and its Implications for Experiential Training Boss/ Subordinate Perceptions of Instrumental and Supportive Leadership Behaviors in Relation to Myers-Briggs Thinking Type Cluster Analyses of American Universities' Business Core Curricula Structures Utilized to Satisfy Fifteen Curriculum Areas Cooperative Learning or Learning to Cooperate Experiential Learning: Constraining Students with Time Budgets How Different Workplace Experiences Affect Different Worker Values Implications of the Trend Toward Relationship Marketing for Experiential Learning The Increasing Cultural Diversity of the American Workforce: Management's Challenge of the 21st Century Information and Uncertainty as Strange Bedfellows: A Model and Experiential Exercises Leadership as a Medium: It's Emergence and Effect on Performance in Small Leaderless Groups Speed, Depth, and Breadth: Assessing Learning in Learning Organizations Teaching Strategic Planning, Problem Solving, and Decision Making with Envisionary Experiential Exercises Validating an Instrument for Student Evaluation of Teachers: Some Noteworthy By-Products Don't Teach Ethics to Business Students Emotional Reactions Toward a Simulated Layoff: Before and After the Manipulation Enhancing Communication Using a Presentation Package Implementing Marketing Policies through a Business Management Simulation Integrating Action-Based Learning into Executive Development Programs On the validity of using the Microsegmentation Principle in Media Simulationsm Participatory Systems Analysis Some Relationships between Cultural, Organizational, and Educational Experience and Perceptions of Influence The Use of Decision Support Systems with a Marketing Simulation: The Future is Now Astute Business Policy: A Simulation of the Automobile Industry The Business Policy Game CEO II: A Gaming Simulation for Assessment Computer Paced Project Management Simulation Computerized Tutor Support Systems DEAL & GEO: Progressively Integrating Gaming Simulations for Entrepreneurship and International Business An Interactive Simulation Game for Competitive Decision-Making International Operations Simulation/Mark 2000 (INTOPIA) Multimedia Simulation Cuts Training Costs for Anderson Consulting Concepts of Total Quality Management: An Active Learning Exercise Cooperative Learning: The Extended Jigsaw Managing Diversity--Values and Attitudes: An Experiential Exercise in Awareness Navigating the Shoals of International Management Development Evaluating Student Performance in the Use of Computer Simulation Entrepreneurial Simulation Program