REVIEW OF LEARNING RESEARCH IN BUSINESS GAMING Computer Simulation and Learning Theory, Volume 3, 1976 173 REVIEW OF LEARNING RESEARCH IN BUSINESS GAMING Bernard Keys--Tennessee Technological University The purpose of this review of learning research in business gaming is to clarify some of the existing incongruencies that exist in the literature. Rather than dealing with very dissimilar learning research studies, this writer has chosen to examine what he considers to be a main-stream of research thrust in business simulation games. Only articles which utilize definite criteria for the measurement of learning and professionally acceptable research techniques are included in the study. This necessarily eliminates articles dealing with such things as the evaluation of performance, attitudes, morale, organizational behavior, and other similar studies which may be of great interest in certain cases. No attempt has been made to choose articles which are favorable or unfavorable toward games, as will be obvious from the research quoted. Instead, articles that tend to lead one toward a general research methodology have been selected. These research studies are outlined in Table 1. In 1962, a classic experiment using the Harvard Business Game was performed by McKenney, comparing a gaming class with a non- gaming class. Both classes of ninety students were tested by having them prepare an analysis of a business case, graded on three planning concepts (6, pp. 115-116). The research design was professionally structured, utilizing an independent grader experienced in case analysis grading and a seven-point scale for each of the three concepts. The class utilizing gaming also utilized cases while the non-gaming class utilized four additional production planning cases in lieu of the game activities. Participants in the game class proved significantly better on two of the planning concepts. According to McKenney, “This comparative study supported a trade of game time for case time, if planning as defined by concepts (2) and (3) is one of the teaching objectives of the course.” (6, p. 116) It is significant that McKenney did not say that the experiment supported a complete substitution of gaming for cases. A more extensive experiment was conducted in 1962, utilizing nine concepts as a grading criterion and a multiple-choice case exam, with the hope that. it would provide more objectivity. While the first experiment seemed to be biased in favor of the case method-since case analysis grading was used for both “game-case” and “all-case” classes--the second experiment chose precepts “. . .for which a game might provide a better learning experience than the case method of instruction.” (6, p. 29) The precepts chosen for the second experiment included five planning concepts and four organizational behavior concepts. Computer Simulation and Learning Theory, Volume 3, 1976 174 Computer Simulation and Learning Theory, Volume 3, 1976 175 Computer Simulation and Learning Theory, Volume 3, 1976 176 Computer Simulation and Learning Theory, Volume 3, 1976 177 Unfortunately, multiple choice examinations were inadequate to allow discrimination among the two classes. The experimenters concluded that “. . .a graduate student is able to create consistent plans in a multiple choice test almost without a reference to the basic data.” (6, p. 117) A rigorous research design was utilized by Raia, in which he compared learning by matched groups of students, including 139 graduating seniors, utilizing 3 instructors and a graduate assistant. Three groups were organized--one group utilizing the case-analysis method coupled with related readings (NG), a second game group utilizing a simple game in addition to cases and readings (SG), and the third group employing a complex game, cases, and readings (CS). However, the game called complex is no more complex than the simpler games being used today where 18 specific decisions per period are incorporated. By interchanging students among groups, the NG, SG, and CS groups were balanced on the variables--assigned instructor, instructional method, grade point average, course load, age, and work experience. Each section covered the same cases at approximately the same time and used the team approach in analyzing them. Students were asked not to discuss their particular teaching aid with members of either of the other two groups. The primary teaching device used in class for all sections was the case analysis method. Students in the experimental groups participated in one of the two computerized games. Students in the control group were assigned a comparable work load of selected readings, each of which required a written review and critique. The outside activities, or experimental period for all groups, covered approximately ten weeks in the middle of the semester. Each week the game groups submitted from two to three decisions, involving decisions in sales, production, and finance. Criteria selected for evaluation included knowledge and skills, interest and motivation, and attitude. Knowledge and skill were tested by evaluating written examinations of students in the various groups both before and after the management game play. Special precautions were taken to insure unbiased grading, i.e., independent graders, re-grading, random grading, etc. Questionnaires were used to measure attitudes and levels of interest, while measures of interest and involvement were evaluated by observation. The written examination consisted of a short business case analysis administered before game play and a different case administered at the end of game play. A separate final exam of multiple choice questions requiring explanations supporting the choice. The comparison of pre- and post-test scores for all three groups (simple game, complex game, and no game) indicated no significant differences in learning. A final examination consisting of true-false questions requiring a supporting choice and questions calling for the application of break-even analysis and sales forecasting. Analysis of the mean Computer Simulation and Learning Theory, Volume 3, 1976 178 scores obtained on the final examination revealed significantly higher scores for the game playing groups, with neither game group scoring higher than the other. Raia concluded that “... when used as a supplementary teaching aid, the games enhanced learning and heightened student interest and motivation, and that relatively simple games promoted essentially the same benefits as the more complex ones.” (8, p. 352) “Perhaps the most significant finding of the study is that game complexity (as defined herein) was not found to be directly related to the benefits that were derived from games. In terms of opportunity costs for faculty and students, participation in a relatively simple game may be more worthwhile than participation in one that is considerably more complex.” (8, p. 352) In a ranking of preference for types of teaching techniques, all three groups, no game, simple game, and complex game, preferred a combination of cases and game methodology. Greenlaw and Wyman suggest this point as being crucial. They suggest that the important question is not whether games are more important as teaching tools, but “. . .rather, whether games in combination with other educational methodologies are the most effective ways to teach certain courses.” (5, p. 276) They conclude “. . .we strongly suspect an affirmative answer to this last question but have no ‘hard’ research to support this conjecture.” (5, p. 276) Greenlaw and Wyman (5) reviewed a massive study by Strother, et. al. (9) reporting several confusing experiments. The test used was a 113-item fact and concept test measuring learning in a simple game type of class compared with learning in a non-game class and a complex game class compared with learning in a non- game class. Apparently, these reviewers could not determine which game was being played in spite of the fact that the report consumed 244 pages (5, p. 272). It also appears that no pre-test measure of learning was used. The researchers concluded “...al- though there were a few cases where significant differences could be found, there was not enough evidence to reject the hypothesis.” (5, p. 272) In a similar study, Strother evaluated undergraduates utilizing a complex game compared with matched non-game classes on the same 113 questions. The fact and concept test yielded four questions on which game learning was significantly better than non-game learning, but the researchers found these differences “unexplainable.” In a trial study evaluating graduates in a complex game class compared with those in a non-game class, Strother, et. al. found four questions which had significantly superior answers in the game class--again items which they found “...difficult to rationalize.” Greenlaw and Wyman chose the following quote as representative of the researchers’ views regarding all three studies (5, p. 274): “...planning and integrating the various functions of business was the predominant awareness which the participants Computer Simulation and Learning Theory, Volume 3, 1976 179 expressed (and) in all cases...large and small teams, without massive effort on the part of faculty members only, an awareness of problems develops rather than an analytical ability to handle them.” (9, pp. 168-169) Moore compared matched classes in a junior-level production management course with one group participating in five production management games and a control group utilizing case studies. Five cases were developed, each corresponding to the problems dealt with in each of the games. According to Moore “. . .an effort was made to present, in case form, the exact subject matter covered by each game. Graphic and tabular illustrations, terminology, and phraseology were transferred intact where possible into the case format. Both groups spent two-thirds of their weekly class time in joint lecture sessions with a single instructor in order to standardize the lecture. For the weekly game, case-session instructors were rotated to avoid systematic bias. Both game and case groups were tested at the last recitation period on a four-part exam. Part I of that exam attempted to measure fact mastery by means of multiple choice and short answer questions. Part II consisted of three essay-type questions to evaluate the student’s ability “...to express detailed concepts in an explicit manner.” Part III required the student being tested to define four terms in order to appraise his “...acquisition of general and structural learning not specifically or directly referred to in the case-game subject matter.” (7, p. 19) Students were also given three cases to complete in outline form outside of class--a test used to evaluate logical reasoning ability. Evaluation was conducted by combining the results of all tests to gain a measure of overall learning. The case group on one of the two campuses examined on fact-mastery only attained a significantly higher mean score on the individual tests. Overall learning proved to be significantly greater for the case group at the other campus. Moore concludes that •his “...study points out the possibility that players of a business game are not strongly motivated to learn about the basic concepts underlying the game. Instead, they tend to become preoccupied with ‘beating the system’ or with competing against each other. On the other hand, the case method, which is based on a relatively static situational setting, may provide an atmosphere more suitable for the examination of key variables and issues related to the subject matter being presented.” (7, p. 22) Evered and Pearce (3) investigated the effectiveness of simulation gaming and case analysis methods in facilitating the acquisition of skills in developing corporate strategy. Effectiveness was measured by means of nine business strategy criteria cited by a well-known policy text. A group of students in one class was Computer Simulation and Learning Theory, Volume 3, 1976 180 taught corporate strategy by means of eight sessions of case analysis, one identical experiential group was taught by means of eight sessions of simulation gaming, and a control group received neither type of instruction. After the sessions both groups were asked to develop a strategy statement from a written description of a complex problem. Strategy statements were scored for quality on the nine criteria by independent judges. This experiment was unique since the primary research instrument used was an interactive, incident style of exercise containing a one-page description of a real-world strategic dilemma. The researchers describe it as follows: The student was instructed to write a statement of his preferred corporate strategy. The one-hour exercise was interactional in the sense that the student had the opportunity to select the items of additional information which he considered necessary to developing his strategy statements. A maximum of 12 items from a set of 25 could be obtained upon request, at four scheduled times (10 minute intervals) with 3 pieces of information per pick-up (3, p. 410). Two independent judges rated the strategy statement of each student on a five-point scale for the extent to which each of the nine criteria in Table 1 were met. Inter-rater and intra-rater reliabilities were found to be quite high. The case group was discovered to be significantly superior to the game group on three criteria, while the game group was not found to be significantly superior on any of the criteria. Both case and game groups showed significant improvement over the “untreated” control group. The writers qualify their findings by pointing out that the game used was a relatively old game (Circa 1961), the UCLA #3, while the cases used were from a modern policy book. Furthermore, the period of time during which participants were exposed to pedagogy was relatively brief--perhaps too brief to account for “learning curves” of the different instructional elements. The researchers could not determine whether the research instrument was adequate in soliciting the depth and breadth of increased skills developed by such instructional techniques. Most importantly, the criteria for evaluating a corporate strategy statement was developed by theoreticians whose primary orientation was toward case analysis. “The possibility exists therefore, that a set of criteria developed from a business game simulation orientation might have yielded results more favorable to the simulation gaming pedagogy.” (3, p. 413) Utilizing Louis Boone’s Marketing Strategy, Fritzsche (4) measured learning in an experimental junior level marketing class, compared to a control group utilizing the same game as an outside activity. The control group used two class sessions for an introduction of the game with all subsequent activity being performed outside the Computer Simulation and Learning Theory, Volume 3, 1976 181 class. Teams in both control and experiential groups developed objectives and strategies and made end-of-quarter written and oral reports. Both control and experimental groups were requested to purchase a text and readings book and were given a copy of a readings schedule. The control group was required to read the material prior to class time, while the experimental group was informed that they would be held responsible for the material on examinations. Students from both classes were provided a list of concepts necessary to satisfy the requirements of the course. The control group sessions were conducted in traditional lecture- discussion activity while the classroom time for the experimental group was spent in game activity and briefing their instructor in his office once weekly. The briefing was a 15-minute session in which students explained their accomplishments over the past week or two game-quarters of play. The sessions included probing questions by the instructor regarding the reasons for actions taken. Classes were used to answer any questions and to allow game decisions to be made, collected, and dispersed. Traditional essay examinations were given both groups at mid-term and final-exam periods and were randomized to prevent grader bias. The experimental group scored significantly higher on both mid-term and final exams. The findings of this experiment must be qualified by the fact that no pre-test measure of student knowledge was utilized. The experimental group also included significantly more full-time employed students than the control group. Cumulative grade averages were not significantly different for the two groups. Wolfe (10, p. 293) compared a game-only business policy class with a case-only business policy class on nine precepts similar to those cited by McKenney earlier. The experiment was carefully designed so that grader bias would be minimized. The major difference in this experiment and the McKenney experiment is the elimination of case assignments from the experimental group. Wolfe concluded that the management game produced learning outcomes that were equal to cases in fact mastery and superior in principle mastery and overall results. Games alone were concluded to be superior to cases alone in the teaching of the measured business policy principles. Wolfe qualifies his experiment by suggesting that game success is probably affected by the quality and degree of teacher guidance and the choice of the game used. In a second parallel experiment, Wolfe attempted to compare a pure experiential class with a mixed case-game approach in the business policy course. The amount and type of knowledge was measured by using the first six questions of the business policy exam used by Wolfe in his first experiment based on concepts developed by McKenney (6) and Cohen (1). Each examination was scored on principle mastery and more specifically, the ability to illustrate the principle correctly. Computer Simulation and Learning Theory, Volume 3, 1976 182 One notable change was made in the research design from Wolfe’s first experiment-the role of the instructor was largely passive, whereas in the former experiment the instructor had guided and structured the learning experience. (In McKenney’s experiments the instructor maintained an active role.) Wolfe found there was no improvement on pre- and post-test scores for the experiential class and concludes that “. . .knowledge increased on only two of six concepts, suggesting that teaching inputs in addition to a game and cases must be employed if an understanding of the four remaining concepts is to be acquired.” (11, p. 449) The author seems to feel that some learning did take place in the experiential group, but the examination used was biased in favor of the cognitive and formal aspects which in theory are not well taught by experiential methods. According to Wolfe “... the question here is whether experience is the best teacher’. Experiential knowledge is so unstructured that student knowledge may be fragmentary and haphazard. Additionally, there is no assurance that the student learns the right thing.” (11, p. 450) A unique study was conducted by Wilson and Wollman in 1975, when they examined the hypothesis that the inconclusiveness in game learning research has resulted from the failure to recognize the different types and levels of learning. Therefore, a test instrument and procedure was designed that “...would measure the influence of simulation exercises at the application or analysis of learning.” (12, p. 1) MBA students in a business policy course were tested by a pre-, post-one, and post-two sequence to study the students’ abilities to identify and solve problems. Control groups participating in traditional policy classes (presumably using cases) were evaluated and compared with classes participating in a simulation. Upon completion of the first post-test the control group also participated in the simulation. During the period between post-test one and post-test two, the game sections participated in other class activities but were re-tested to determine if any decay in skills took place. The unique aspect of this research was the way in which the pre- and post-test was developed. Test situations containing “...one-half to one page of text material alluding to the possible areas and three pages of statistical data were presented in a more traditional format than the result from the simulation.” (12, p. 4) Economic and seasonal factors were changed and students were not told that the data was derived from the game. According to the researchers, the participating (game) sections performed significantly better (.001) in pre- and post-one tests than the control groups who had not participated in the simulation exercise. Also post-one and post-two results were significantly better than pre-test results. Not only did students playing the simulation identify more problems than the control group, they also improved in their abilities to identify problems. Computer Simulation and Learning Theory, Volume 3, 1976 183 The evaluators found no significant difference in the quality of the solutions between the simulation groups and the control groups, indicating that the simulation teams did not improve in their abilities to solve problems. No description is given of the non-game class activities or of the role of the instructor in the game classes. CONCLUSIONS 1. Cases and game were found to be superior to cases alone in a business policy course. (6) 2. A multiple choice or objective test is not subtle enough to capture game or case-type learning. (6, 8) 3. In terms of faculty student time tables, a simple game and cases may be more worthwhile than a complex game and cases. (8) 4. A case-type class is superior to a game-only class in teaching business policy, corporate strategy, and production management fact mastery. (3, 7, 9) 5. There is some indication that game time can be effectively substituted for lecture time in a marketing class, when similar game decisions, objectives, and readings are assigned both classes. (4) 6. A game-only business policy class produces superior results to a case-only business policy class where significant instructor guidance is provided in the game-only class. (10, 11) 7. A case-game class is superior to a game-exercise class in teaching business policy concepts when the instructor remains relatively passive in the game- exercise class. (12) When the total teaching mixture is examined in each of the studies reviewed, with the role of the instructor carefully scrutinized, there appears to be no conflict in the findings. In teaching business policy concepts, cases, and games appear to be superior to cases alone, games alone, or games plus exercises, if the instructor is allowed to play an active and significant role in guiding game learning. In addition, given a reasonably effective blend of game complexity, cases and instructor guidance, course time can usually be made more effective by trading reflective learning, assignments, and discussions for game complexity. There is some indication that functional courses can afford to trade class lecture time for game activities and still produce the same effectiveness on essay-type exams. REFERENCES 1. Cohen, Kalman J., et. al., “The Carnegie Tech Management Game,” The Journal of Business, Vol. 33, No. 4 (1960), pp. 303-321. Computer Simulation and Learning Theory, Volume 3, 1976 184 2. Dill, William R., and Neil Doppelt, “The Acquisition of Experience in a Complex Management Game,” Management Science, Vol. 10, No. 1 (1963), pp. 30-46. 3. Evered, Roger D., and John A. Pearce, “Business Simulation Gaming as an Alternative to Case Analysis for Teaching Corporate Strategy--Formulation Skills,” Proceedings of Thirteenth Annual Symposium, National Gaming Council, Vol. 2, (1974), pp. 309-413. 4. Fritzsche, David J., “The Lecture vs. The Game,” Simulation Games and Experiential Learning Techniques: On the Road to a New Frontier; Proceedings of a National Conference on Business Gaming and Experiential Learning (Norman, Oklahoma: The Center for Economic and Business Research, University of Oklahoma (1974), pp. 41-46. 5. Greenlaw, Paul S., and F. Paul Wyman, “The Teaching Effectiveness of Games in Collegiate Business Courses,” Simulation & Games, Vol. 4, No. 3 (September 1973), pp. 259-294. 6. McKenney, James L., Simulation Gaming for Management Development, The Division of Business, Harvard College (1967). 7. Moore, Larry F., “Business Games vs. Cases as Tools of Learning,” Training and Development Journal, Vol. 21, No. 10 (1967), pp. 13-23. 8. Raia, Anthony P., “A Study of the Educational Value of Management Games,” The Journal of Business, Vol. 39, No. 3, (July, 1966), pp. 339-352. 9. Strother, E., et. al., “Educational Application of Management Games,” NDEA Title VII Grant, 7-59-0500-252. 10. Wilson, Harold K., and Fred A. Wollman, Jr., “The Effectiveness of a Computer Simulation Exercise in a Business Policy Course,” submitted to Teaching Methods Section, American Institute of Decision Sciences, Seventh Annual Conference (November, 1975). 11. Wolfe, Joseph, “The Comparative Learning Effects of a Management Game vs. Casework in the Teaching of Business Policy,” Academy of Management Proceedings 33rd Annual Meeting (1973), pp. 291-298. 12. Wolfe, Joseph, “A Comparative Evaluation of the Experiential Approach as a Business Policy Learning Environment,” Academy of Management Journal, Vol. 18, No. 3 (1975), pp. 442-452.