A DEMONSTRATION OF THE BUSINESS SIMULATION-GAME AS A CURRICULUM ASSESSMENT DEVICE Insights into Experiential Pedagogy, Volume 5, 1979 191 A DEMONSTRATION OF THE BUSINESS SIMULATION-GAME AS A CURRICULUM ASSESSMENT DEVICE Joseph A. Wolfe, The University of Tulsa ABSTRACT It has been suggested that a business game can serve as a curriculum evaluation device. This paper outlines the method and results of a demonstration of this type of simulation application. All teaching inputs were held constant over three different business colleges where all subjects had completed their business college core requirements. Differential performance results were obtained and certain departmental quality differences were perceived and contrasted on a comparative basis. The demonstration’s results generally reflect admissions criteria and other indices of business school excellence. INTRODUCTION Various business-gaming proponents in the early 1960’s foresaw and charted much of what wag to become simulation’s most productive and influential areas. Cohen and Rhenman [6] noted gaming’s then-current use for business education and its future applications as social science laboratories and research environments for a wide variety of disciplines. Dill [8] and Dill, Jackson and Sweeney (9] , while restricting their vision to the educational sphere, still suggested an ambitious array of business- gaming applications. Whether applied to executive development programs [8] or to business schools in general [9] , it was believed that games could be used to stimulate, simulate, teach and test. The prospects and realities of the “stimulate, simulate, teach aspects have received a great deal of vigorous debate and research attention but the ‘test’ aspect has remained largely fallow and untapped. To Cohen and Rhenman, testing entailed the use of business games for career guidance or executive assessment [6, p. 165]. Inferential work along this line has been performed by Babb, Leslie and Van Slyke [1], McKenney and Dill [21] , Vance and Gray [27] , and Wolfe [29] while direct work has been reported or performed by Baldwin [2], Cohen and Rhenman [6], Hunsaker [15; 16], Messmer [22] , and Schreier and Komives [23]. This paper, however, is more concerned with Dill, Jackson and Sweeney’s [9, p. IL] use of games for testing not so much to grade students as to determine whether the educational program which they had undergone had satisfactorily achieved its objectives.” In this curriculum-evaluation sense, testing emphasizes the institution’s performance rather than the individual student’s performance in a business game. Given this focus, an institution’s game performance could be used (1) within a business college to evaluate certain aspects of its departmental academic output, and/or (2) between institutions to help judge the nature and quality of their overall programs. METHODOLOGY It has become common practice to use national tests to evaluate students, judge the quality of the programs to which they have been exposed, and to “level” the differential grading scales that have been applied to them. Standardized tests such as the ACT, GRE, SAT and GNAT have been used as instruments for college admissions decisions [7; L2; L3; 18], the prediction of professional accomplishment [7; 20], and the evaluation of an academic program’s quality [Li; L9; 25]. In the study reported here, a comprehensive, computer-based business decision-making simulation [17] became the instrument through which three different business college degree programs received an external and internal evaluation. To serve as an unbiased evaluation instrument on both the individual- student and macro-organizational level, It is necessary that the gaming/instrument be capable of testing the respondent’s knowledge along the following dimensions: 1. comprehensiveness or variety; 2. depth or assortment; and 3. factual, theoretical and/or skill orientation. The simulation employed in this study had been found to be both comprehensive [34] and functionally and cognitively unbiased (30; 32; 34] ; it had also been able to sustain motivation and interest over a comparatively long period of time thus insuring deeper and more stable measurements of knowledge [34]. Complex simulations of this type also possess high player face validity and legitimacy as the games have been found to reward rational decision- making behaviors while simultaneously serving as motivating and compelling learning experiences [1; 9; 29; 31; 33; 34; 35]. This study’s simulation had previously demonstrated additional internal academic validity characteristics. Wolfe [30] had found that students could meaningfully express and translate their knowledge into profitable forms within the simulation’s context where correlations between game profits and various cumulative grade- point-averages (CGPAs) ranged from .373 to .503. This compares with ATGSB score correlations and first-year MBA grades which typically range from .43 to .53 [13, p. 97] , CACT scores and overall college grades which are...about .50” [28, p. 614] , or SATs and high school grades which hove consistently held at about .55 [10, p. 647]. Three midwestern schools were the focus of this study and have been profiled below. Each institution’s CACT score and Barron’s [3] classification of entrance competitiveness has also been included. Alpha A small, private 2,600-student, liberal-arts, dormitory- type school. Alpha is situated in a fairly affluent suburb of a major midwestern city and offers a very active part-time evening program in business administration or accounting. The school recently faced a financial scare which has caused it to (L) emphasize the more-practical and job-oriented disciplines, (2) de- emphasize Liberal Arts and (3) actively seek full-time black students to shore up deteriorating full-time day enrollments. Accepts about 91% of its applicants of which 51% actually enroll; 17% of its Freshmen scored over 25 on the CACT and 15% of its Freshmen were in the upper 10% of their high school classes. CACT = 24, Barron’s = Competitive. Delta A 19,000-student, comprehensive, non-dormitory state Insights into Experiential Pedagogy, Volume 5, 1979 192 university located in a major American city. Minority- group members are becoming an increasing proportion of the student population. Almost one- half of its graduates transfer-in from nearby colleges. Delta’s business college faculty possesses both size and depth, and which, while basically young, has been recruited from the nation’s top doctoral programs. The college accepts about 67% of its applicants and 78% of those accepted enroll. 21% of its Freshmen were in the top 102 of their high school classes. CACT 22, Barron’s Very-competitive. Sigma A medium-sized, 12,300-student, comprehensive, non- dormitory urban, university. Sigma has changed its governance from public/municipal to public/state control and is in the process of up-grading its faculty and facilities since receiving fresh State university- system money. Salary levels are somewhat depressed and the faculty has recently voted in the AAUP as its collective bargaining agent. The school has an open admissions policy where 72% of all admitted elect to enroll. CACT = 19.6, Barron’s Non- competitive. 18% of its Freshmen score over 25 on the CACT. Business administration seniors played the simulation game for credit within a core-required Business Policy course. All teaching inputs were held as constant as possible across all institutions where the instructor, course-level, textbook, reading assignments, simulation, and case assignments were all controlled. The measurement of individual game achievement (to mirror individual achievement as tested in the ACT/SAT/GMAT situation) was accomplished by creating single-member firms in 8-firm industries. Both quantitative and qualitative measures were recorded. Quantitative measurements were in the form of cumulative earnings (CEARN), rate-of- return on investment (ROI) and rate-of-return on equity (ROE). Qualitative measures were taken of the students perceptions regarding the adequacy of their prior coursework and the theory or skill orientation of that coursework. RESULTS The Mann-Whitney U test was employed to determine if differences existed between the performances obtained by each university’s students. As presented in Table 1 there were non-significant differences between the profits and rates-of-return generated by and Delta while Sigma underperformed both at the .001 level. Alpha's earnings ranged by firm from $302,254 to $-498,563 with Delta’s ranging from $330,262 to $- 383,209. None of Sigma's firms ever generated overall profits where the best-performing firm lost $-160,504 while simultaneously obtaining a - 7.96% ROE. Table 2 presents the Likert-type questions used to obtain the perceptual and qualified data used in this study. As presented, the questions have been transformed slightly as the original instrument alternatively posed its questions in a positive/negative sequence. Questions 1-3 found no overall differences as to perceived coursework adequacy despite real performance differences obtained by Sigma. This result was possibly due to the nature of the questions’ form. As shown by the F-test conducted on Table 3’s data, response differences did exist when the same basic question was addressed on a departmental basis. Questions 1-3 were possibly tapping an algebraic sum of evaluations where the strengths of one department’s offerings were being counterbalanced by the weaknesses of another’s [4; 5]. Question 4 found that no school or department offered 1 1 Median values displayed for comparative purposes. Mann-Whitney U test for significant differences in ranked arrays. Insights into Experiential Pedagogy, Volume 5, 1979 193 2 an academic focus that was differential as to its technical/skill/theoretical orientation. Given this result, each school’s departmental score was averaged between its technical and theoretical orientation to produce an overall score regarding the degree each department’s courses prepared the student for successful game play. Table 3’s F-test found that different departmental variances existed between programs and that each school prepared its students differently for the simulation on a departmental basis. In Sigma’s case, the within- school departmental differences were so extreme that statistically-significant differences were found. Based on measures of perceived game preparedness, Sigma’s Management department was ranked highest; the Marketing, Economics, Accounting and Quantitative Methods departments occupied the second rank; the Finance department occupied the lowest rank. Table 4 presents the same data after being subjected to an inter- organizational, across-department analysis.• Given the quality of preparedness afforded by previous departmental coursework, Sigma’s Finance and Economics departments were inferior to Alpha’s and Delta’s while all Management departments were of equal quality. Delta’s Marketing, Accounting and Quantitative Methods departments were superior to Sigma's; no statistical statement can be made regarding Alpha’s respective departments. Overall, Alpha and Delta’s departments were perceived to be of equal value and while Sigma’s was inferior to both. 2 All mean differences within programs non-significant unless otherwise noted. 3 DISCUSSION In discussing the business simulation as a curriculum evaluation device, it is best to delineate those aspects which are unique to simulations as evaluators versus those aspects that generally apply to all evaluation instruments. Turning first to evaluation instruments in general, no device employed In an ‘after-only sense without a pre-program measure of knowledge could determine how well the institution performed given the quality of students it was able to enroll. It would also be practically impossible to design an instrument that could tap all the elements that make up a college education. Accordingly, any instrument is only a partial barometer of an institutions output. Additionally, no instrument can determine the sufficiency or adequacy of a program without some type of external reference point. Within the context of this demonstration, it was methodologically implied that each university’s students perceived and reacted to the simulation and the qualified examination in a uniform fashion. A partial test for this disqualifying factor was made In Question 5 and 6. In this case all students, regardless of the institution, rated the game and the course equally as a well-rounded, challenging, and useful experience. Respective game and course scores were 4.41 and 4.21 on a 5.0 scale. It must also be assumed that the students enrolled in the simulation were a representative sample of the university’s output. During the two years covered by this study, the instructor’s class enrollments represented 3 Average of technical and theory scores. All differences between means across programs non-significant unless otherwise noted. Insights into Experiential Pedagogy, Volume 5, 1979 194 73.1, 44.9 and 38.0 percent of those schools’ Business Policy students. Demographic information collected on the enrollments found no significant within-differences between those enrolled in the experimental and non- experimental sections by age, sex, academic major, GPA, and hours of outside employment. Significant demographic differences did exist between institutions and these differences might have been presumed from the profiles presented earlier. It could be expected that glaring differences would exist between Delta and Sigma given the Barron’s label of very-competitive” versus non- competitive.” Student self-selection decisions and admissions officers’ acceptance criteria reflect the (1) give-and-take and relative degrees of competitiveness for students, (2) adequacy of the high school student’s preparation, and (3) presumed degree of rigor that can be demanded of the students during their college years. Sigma, which accepts 100% of its applicants, is in turn rejected by 28% of those it accepts. This could indicate Sigma’s relative weakness in the market for highly-qualified and geographically mobile students. One should wonder, however, why the simulation did not detect similar differences between Alpha and Delta. which generates a relatively-low “yield rate” of 49% (the ratio between eventual’ student enrollment and applications accepted), and possesses a relatively unspecialized business faculty in a liberal arts-dominated school, could be expected to underperform Delta’s product [261 . It could be surmised that Alpha was performing a superior job on possibly inferior students, or that its (mostly part-time) students had augmented their academic studies with valuable life- experiences which were being expressed in the simulation [29]. Conclusion This paper has described the use of a comprehensive business game as an inter- and intra-organizational curriculum device. Certain quantified and qualified program differences were noted. It is still a matter of conjecture as to whether (1) one could have more simply reached the same conclusions on an inter- organizational basis from publicly-available college directories, or (2) a comprehensive simulation of the type used here measures all that an educator would wish a student to know after four years of college. More positively, the simulation appears to be discriminating at the intra- organizational level of analysis. Statistical differences were found in the quality of Sigma's departmental offerings and more cogent instruments could have possibly pinpointed the specific sources of program variance detected at Alpha and Delta. REFERENCES [1] Babb, E. N., M. A. Leslie, and M. D. Van Slyke, The Potential of Business-Gaming Methods in Research,’ Journal of Business, Vol. 39, No. 4, 1966, pp. 465-472. [2] Baldwin, L. E. Student Self-Development Planning From Simulation Experience Inputs,” in Burnard H. Sord (editor), Computer Simulation and Learning Theory, 1977, pp. 240-248. [3] Barron’s Profiles of American Colleges, Vol. 2, (Woodbury, N.Y.: Barron’s Educational Series, Inc., 1973). [4] Bettman, James R. Data Collection and Analysis Approaches for Studying Consumer Information Processing,” Working Paper No. 41, Center for Marketing Studies, University of California, Los Angeles, July 1976. [5] Bettman, James R., Noel Capon, and Richard J. Lutz, ‘Cognitive Algebra in Multi-Attribute Models,” Journal of Marketing Research, Vol. 12, No. 2, 1975, pp. 151-164. [6] Cohen, Kalman J., and Eric Rhenman, “The Role of Management Games in Education and Research,” Management Science, Vol. 7, No. 2, 1961, pp. 131-159. [7] Creager, John A. “Predicting Doctorate Achievement with GRE and Other Variables,” Technical Report No. 25 (Washington, D.C. National Academy of Sciences, November 1965). [8] Dill, William R. “What Management Games Do Best, Business Horizons, Vol. 4, No. 3, 1961, pp. 55-64. [9] Dill, William R., James R. Jackson, and James W. Sweeney, Proceedings of the Conference on Business Games, (New Orleans: Tulane University, 1961). [10] DuBois, Philip H., in Mental Measurements Yearbook, Vol. I, Oscar Krisen Buros (ed.), (Highland Park, N.J.: The Gryphon Press, 1972), pp. 646-648. [11] Educational Testing Service, Survey of College Achievement, (Princeton, N.J.: Educational Testing Service, 1969). [12] Graduate Record Examinations: National Program for Graduate School Selection, (Princeton, N.J.: Educational Testing Service, 1970). [13] Handbook for Deans and Admissions Officers, Admission Test for Graduate Study in Business, (Princeton, N.J.: Educational Testing Service, 1966). [14] Henkel, Jan W. The Educational Background of New CPA’s as a Predictor of Success on the Uniform CPA Examination,” Collegiate News and Views, Vol. 31, No. 3, 1978, pp. 5-7. [15] Hunsaker, Phillip L. ‘Leadership Evaluation and Training Through Behavioral Simulations: Method, Results, and Future Directions,’ in Carl C. Nielsen (editor), New Horizons in Simulation Games and Experiential Learning, 1977, pp. 40-47. [16] Hunsaker, Phillip L., W. C. Mudgett, and B. E. Wynne, “Assessing and Developing Administrators for Turbulent Environments,” Administration and Society, Vol. 17, No. 3, 1975, pp. 312-327. [17] Jensen, Ronald L., and David J. Cherrington, Business Management Laboratory, (Dallas: Business Publications, 1977). [18] Kelley, Truman L., Richard Madden, Eric F. Gardner, and Herbert C. Rudimen, Stanford Achievement Test, N.Y.: Harcourt Brace Jovanovich, 1968). [19] Kellogg, T. E., Paul L. Trump, Richard Pearson, Robert L. Foose, Loren Benson, and Byron L. Grosbeck, “National Admissions Testing Programs-Their Value to Colleges--Their Impact on Secondary Schools, College and University, Vol. 39, No. 4, 1964, pp. 488-514. [20] Lewis, John W. ‘The Relationship of Selected Variables to Achievement and Persistence in a Master’s Program in Business Administration,” Educational and Psychological Measurement, Vol. 24, No. 4, Insights into Experiential Pedagogy, Volume 5, 1979 195 1964, pp. 951-954. [21] McKenney, J. L., and W. R. Dill, “Influences on Learning in Simulation Games," American Behavioral Scientist, Vol. 10, No. 3, 1966, pp. 28-32. [22] Messmer, Donald J. “A Simulation Enhancement to Observe, Motivate and Evaluate Individual Performance,” in Burnard H. Sord (editor), Computer Simulation and Learning Theory, 1977, pp. 358-371. [23] Schreier, James W. and John L. Komives, “Assessment of Entrepreneurial Skills Using Simulations,” in Burnard H. Sord (editor), Computer Simulation and Learning Theory, 1977, pp. 53- 60. [24] Strickland, A. J. ‘The Use of a Management Simulation as a Predictor of Success in Graduate Studies, Paper presented, Association for Business Simulation and Experiential Learning, Denver, Colorado, April 1978. [25] Student Profile Report, (Iowa City, Iowa: The American College Testing Program, 1970). [26] This University Wants You!” Time, Vol. 111, No. 22, May 22, 1978, p. 69. [27] Vance, S. C., and C. F. Gray, Use of a Performance Evaluation Model for Research in Business Gaming,’ Academy of Management Journal, Vol. 10, No. 1, March 1967, pp. 27-37. [28] Wallace, Wimburn L. in Mental Measurements Yearbook, Vol. I, Oscar Krisen Buros (ed.), (Highland Park, N.J.: The Gryphon Press, 1972), pp. 612-615. [29] Wolfe, Joseph, Correlates and Measures of the External Validity of Computer-Based Business Policy Decision-Making Environments,” Simulation & Gaines, Vol. 7, No. 4, 1976, pp. 411- 433. [30] Wolfe, Joseph, ‘Correlations Between Academic Achievement, Aptitude, and Business Game Performance,” in David C. Brenenstuhl and Samuel C. Certo (editors), Exploring Experiential Learning: Simulations and Experiential Exercises, 1978, pp. 316- 324. [31] Wolfe, Joseph, “Effective Performance Behaviors in a Simulated Policy-Making Environment, Management Science, Vol. 21, No. 8, 1975, pp. 872- 882. [32] Wolfe, Joseph, “Learning Styles Rewarded in a Complex Simulation with Implications for Business Policy and Organizational Behavior Research,” in Robert L. Taylor, et. al. (editors), Academy of Management Proceedings, 1976, pp. i64- 168. [33] Wolfe, Joseph, “The Effects and Effectiveness of Simulations in Business Policy Teaching Applications,” Academy of Management Review, Vol. 1, No. 2, 1976, pp. 47-56. [34] Wolfe, Joseph, “The Effects of Game Complexity on the Acquisition of Business Policy Knowledge,’ Decision Sciences, Vol. 9, No. 1, 1978, pp. i43- i55. [35] Wolfe, Joseph, and Gary Guth, “The Case Approach vs. Gaming in the Teaching of Business Policy: An Experimental Evaluation, The Journal of Business, Vol. 48, No. 3, 1975, pp. 349-364. Table of Contents Volume 6, 1979 The Use of Cases with Role Plays in a Research Methods Course Debugging and Implementing the Live-Case Approach to Marketing Research in the Australian Environment Using a Case as the Basis for a Modified Debate Using Cases in Business Communication Classes A Structured Approach to Case Analysis and Reporting Intercollegiate Case Analysis Competition as an Experiential Learning Experience Attributes Germane to Student/Live Case Situations Understanding Dispute Resolution Through Experiential learning Personality Development and Conflict Dynamics: An Experimental Design to Study the Effects of Teaching Methodologies on Conflict Resolution Interpersonal Competence and the Digital PDP-11:40 The process of Writing a Collective Bargaining Simulation: A Case Study in Practical Pedagogy Simulating Negotiations in an Educational Context Simulations as a Technique for Teaching Collective Bargaining Experiential Processing of Differing Managerial Perspectives: The Use of a Game Show Format Individual Self Report vs. Group consensus in Small Decision-Making Groups Effects of Sex-Role Stereotypes on Promotion Decisions: An Exercise Computer Aids to Planning: The Budget and Forecasting Module Use of a Microcomputer in a Decision Analysis for Investment Portfolio Selection A Game of Investment Strategy: Description, Use, Criticism and Modification Coping with Future Uncertainties Through Probabilistic Budgeting Administration and Design of Simulation Materials for a Specialized Management Training Program Managing Stress in Organizational Life Banksim: The Bank Management Simulation Issues in the Organizational Application of Simulation and Experiential material Games Within Games: The Role of the Glitch Experiential Learning from Classroom to Business An Evaluation of the Small Business Institute Program as an Experiential learning Exercise Conversations with Top Management (Simulated) Experience in the Use and Assessment of Simulation in Management Education A Statistical Analysis of Simulation Users: Relationships between Stress and Human Subject guidelines Group Decision Making in a Computer Game Analysis of Demographic and Psychosocial Variables Comparing Performance During Three Managerial Accounting Simulation Schedules Business Policy Simulation and the Intense Course Structure The Use of Intensive Simulation in Executive Development and Academic Settings The Use of Experiential Exercises in the Undergraduate Consumer Behavior Course A Personal Marketing Strategy Approach: Framework and Application Teaching PERT Experientially in Marketing Research Discovering the Majority Fallacy An Experiential Approach to Studying International Business A Demonstration of the Business Simulation Game as a Curriculum Assessment Device Relating Teaching Methods with Educational Objectives in the Business Curriculum The Junior Achievement Applied Management Program as a Compromise Situation Between the Simulation and the Internship Who is Using Computerized Business Games?: A View from Publishers' Adoption Lists Trials and Tribulations in Testing Educational Innovations An Examination of the Perceived Effectiveness of Computer Simulation in a Classroom Setting as Affected by Game, Environmental and Respondent Characteristics Game Administration: A Life Cycle Analysis Research on the Effectiveness of Using a Computerized Simulation in the Basic Management Course Gaming and Attitudinal Change The Teacher-Student Relationship in Experiential Classes and the Student's Perception of Course Effectiveness An Exploratory Study of Student Characteristics and Educational Processes in Programmatic Experiential learning General Incongruity Adaption Level (GIAL) as a Predictor of Risk Preferences in a Simulated Management Game The Cap-Stone Opportunity: Combining Business Simulation and Experiential Learning Incorporating MIS/DSS into Policy Courses Via Simulation Who Benefits Most from Participation in Business Policy Simulations: An Empirical Study of Skill Development by Functional Areas The Use of Program NAMEX in Teaching the Accounting for Nonmonetary Assets The Use of Program CVP in Teaching Cost-Volume-Profit Analysis Experiential vs. Traditional Classroom Approach in the Basic Management Course: A Puerto Rican Experience A Guide to the Successful Use of Business Simulation Games The Dynamic Aspects of Interactive Gaming Puts the Realism into Gaming System Representation of SIMORG The Design of a Database System to Support Business Simulation and Experiential Learning A Data Entry and Retrieval System for a Computer Simulation (DERS) Simulation - An Advantage for Accounting Research Computer-Aided Project Performance Control Simulation: An Interactive Experiential Gaming Technique for Managerial Decision Making Utilization of Manual Simulation Games to Develop Scenarios of Future Events - An Exploratory Study Foundry: A Foundry Simulation Customized Debriefing: The Achilles Heel of Experiential learning? Computerized Business Simulations and Experiential Learning Exercises: An Instructional Interface The Baseball Game: A Group Role Observation, Problem Solving Experience A Practical Design for Experiential Learning Exercises: Roles, Technical Equipment and Alternative Debriefing Formats