FACTORS AFFECTING STUDENT PERCEPTIONS OF LEARNING IN A BUSINESS POLICY GAME Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 92 FACTORS AFFECTING STUDENT PERCEPTIONS OF LEARNING IN A BUSINESS POLICY GAME Michael Hergert, San Diego State University, San Diego CA Robin Hergert, Southland Software, Coronado, CA A recent survey by Faria (1987) revealed that some 200- business games have been adopted by over 8,500 instructors in approximately 1,900 colleges having business programs. The broad use of simulations has spawned an extensive body of research into their effectiveness as pedagogical tools. (For a comprehensive review, see Greenlaw & Wyman, 1973, and Wolfe, 1985.) Although Faria’s survey also reported that business school deans and teachers using games rated their effectiveness highly compared with other teaching methodologies, most researchers point out that measuring learning and business gamings effects on learning is an extremely difficult task. Wolfe (1985), Greenlaw and Wyman (1973), and Miles et.al.(l986) have noted that the difficulty of stating generalizations about the effectiveness of simulations stems from the variety of classroom practices and differences among games. In this study we have analyzed the impact of a number of variables on student perceptions of simulation usefulness in learning business policy course material. These factors may be organized into four categories: course structure, game parameters, student characteristics, and student effort and performance. Weohave confined our investigation to an analysis of student perceptions for the following reason: Inasmuch as perceptions of effectiveness include components of satisfaction and motivation, and inasmuch as motivation is important to learning, perceived learning may have some correlation Co actual learning. METHODOLOGY AND FINDINGS Data on the determinants of student perceptions of business simulation usefulness was collected through the administration of a written survey (available from authors on request). The survey was completed by 482 students divided into 65 companies in 18 industries. The simulation used was The Business Policy Game by Richard Cotter and David Fritzsche (second edition). The simulation was part of an undergraduate business policy course taught by five different instructors. The game was centrally administered by a faculty member with the assistance of a team of students who had previously played the simulation Students generally found the simulation to be a useful tool for learning the material in the business policy course. Students were asked to rate the usefulness of the simulation on a 5 point Likert scale ranging from 1 (Not at all useful) to 5 (Extremely useful) Students were also asked to compare the usefulness of the simulation to other pedagogical methods, including lectures, case studies, textbook readings, examinations, presentations, and written reports. Finally, students were asked if they would recommend the simulation to a classmate in selecting a section of the business policy course. Although the policy course is required for all business majors, the simulation is not used in every section. The simulation has therefore become one of the decision criteria in choosing which class to take. The overall rating of the simulation was fairly high. This is a similar finding to earlier investigations by Wolfe (1977), and Remus and Jenner (1981), Brenenstuhl (1975), Wolfe and Guth (1975) , Sugges (1982), and Kaufman (1976). When asked to rate the usefulness of the simulation on a five point scale, the mean response was 3.7. On a five point scale ranging from 1 (Would not recommend) to 5 (Would highly recommend), the mean rating was 4.0. Further analysis was conducted to determine what factors account for the favorable rating of the simulation. COURSE STRUCTURE The simulation was used by different instructors with varying emphasis on the game versus other pedagogical approaches. The courses varied in size as well as the weighting placed on the simulation for calculating final student grades. Statistical analysis of these factors showed a significant impact on student perceptions of simulation benefits from varying the course context. Instructors were interviewed about the relative amount of course time spent on the simulation during class meetings. The professors were then scaled from 0 (no class time spent on the simulation- the game was treated as an outside laboratory) to I (the simulation was extensively integrated into the course on a daily basis). This variable was Chen correlated with student ratings of usefulness and the strength of recommendation. The correlation coefficients for usefulness (.136) and strength of recommendation (.135) were both significant at the 99% level of significance. It appears that the game is better received if it can be explicitly linked to course materials. This supports Wolfe and Guths (1975) contention that business simulations can be exciting and relevant teaching tools with structure and support from the instructor. The simulation also fared well relative to other pedagogical approaches. As shown in Table 1, the simulation was regarded as slightly superior to lectures, case discussions, case reports, and textbook readings. Exams, presentations, and other written assignments were considered slightly superior to the use of the simulation. None of these differences were statistically significant at a 95% confidence level. This is consistent with earlier work by Strother et. al. (1966), Raia (1966), Anderson et. al. (1964), Anderson and Woodhouse (1984), Boseman and Schellenberger (1974), Wolfe and Byrne (1976), and Roberts and Field (1975). These findings differ Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 93 somewhat from those of Miles et. al (1986) and Sugges (1982), whose subjects perceived cases as equal or superior to simulations. Wolfe (1977) found a much higher rating of simulations relative to other teaching approaches. The weight placed on the simulation towards final course grade also proved to be a significant determinant of student satisfaction. The weights placed on the game varied between 10% and 33% for the five instructors. The correlation of grade weights with usefulness (.255) and strength of recommendation (.241) are both significant at the 99% confidence level. It appears that placing too small a weight on the simulation can reduce student perceptions of learning, perhaps by undermining their commitment to the game. The role of class size was also investigated to see if the game works better in smaller classes. The correlation between class enrollments and perceived usefulness (.072) as well as strength of recommendation (.055) were not statistically significant. Finally, a correlation analysis was done between student perceptions of simulation benefits and their overall rating of the faculty member as an instructor. Although the correlations between instructor rating and perceived usefulness (.181) and strength of recommendation (.165) are statistically significant at the 99% level, the causality of this relationship is very much open to question and must be interpreted with care. PARAMETERS OF THE CAME In addition to modifying course structure to take maximum advantage of a simulation, and instructor can also vary certain parameters of the game. The variables manipulated in this study were the number of team members per company, the number of worlds (industries) per class, the strength of the economy for the game (as measured by growth in real GNP), the volatility of the economic environment (as measured by the standard deviation in economic growth) and the number of decisions students were asked to make during the semester. Table 2 presents the correlations between the rating of simulation usefulness and the parameters of the game, which were manipulated by the instructor. A similar analysis was performed for the determinants of how likely students were to recommend the game to their peers. The correlations in Table 2 reveal some interesting patterns about student perceptions of simulation usefulness. The only significant variable related to group size. Students rated the simulation more highly if they were on a smaller team. It is likely that smaller group size implies greater responsibility and effort for each individual. The greater effort may produce greater learning and a higher perceived usefulness. Other parameters of the game had a less direct bearing on student satisfaction. The simple correlation coefficients for number of worlds, economic characteristics, and number of decisions were insignificant. However, these variables were analyzed in more depth and shown to be significant when other effects are controlled for. The results of this analysis appear at the end of this section in a regression model of student perceived learning. STUDENT CHARACTERISTICS The perceived benefits of the game may also depend on the individual characteristics of the participants. Data was collected on previous academic achievement (CPA), expected grade in the business policy course, and major field of study. A positive and significant correlation between expected grade and perceived usefulness (.147) and strength of recommendation (.154) was observed, although the fact that the simulation constitutes a part of the course grade makes this relationship a bit difficult to interpret. As will be seen later, students who do well at the game tend to rate it more highly. These students will also expect a higher course grade and it may be incorrect to assume that the simulation appeals to the better students. Further evidence of this is the fact that there is no significant correlation between overall CPA and game rating. Student major was also investigated as a potential determinant of simulation rating. A one-way analysis of variance was run on student major versus perceived usefulness and strength of recommendation. The ANOVA did not detect any difference in simulation rating between students of differing backgrounds. GAME EFFORT AND PERFORMANCE The final area of investigation related to the roles of student effort and success in determining how positively the game was perceived. Variables measuring both individual as well as team effort and performance were collected and analyzed. Students were asked how much time they spent on the simulation each week. The mean for this variable was 5.23 hours per week. However, this variable was not significantly correlated to either perceived usefulness (.092) or strength of recommendation (.083). Students were also asked to submit an organization chart with their title and responsibilities on it. An earlier study by Rowland and Gardner (1973) found that job position in a simulation affected student perceptions of the game. We performed a one-way ANOVA to see if type of responsibilities affected simulation rating. The ANOVA indicated that job title did have a significant affect on simulation rating, so a set of pairwise t-tests were run to investigate what positions in the game seemed most satisfying. The results of this analysis appear in Table 3. As shown in Table 3, students with responsibility for marketing gave the simulation the highest overall rating. Their mean rating was significantly higher at a 95% confidence level. Of course, this conclusion is closely linked to the nature of the specific simulation used in the study. In his analysis of total enterprise games, Keys (1987) found that The Business Policy Game by Cotter and Fritzsche was more marketing oriented than other business games. This may account for the relatively high rankings by students responsible for the marketing function. A similar conclusion can be drawn for the students Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 94 responsible for finance. Keys also concluded that The Business Policy Game did not emphasize financial concerns. This may account for the significantly lower ratings of the simulation by financial managers. It is also interesting to note that the ratings by students who served as GEO/President were not significantly higher than the rest of the participants. Previous studies (Dill and Doppelt, 1963) have found that higher levels of responsibility are generally associated with greater satisfaction. Although the mean rating for CEO/Presidents is higher than for other job titles, the difference is not significant. Team effort and performance were also analyzed to see how they affected simulation ratings. Several authors have found that students who do well in simulations tend to rate them more highly (Dill and Doppelt, 1963, Remus, 1977, Porter and Lawler, 1968), although Dill et. al. (1961) found no relation between game performance and satisfaction with the game. Our sample showed a significant positive correlation between company financial results and both perceived usefulness (.173) and strength of recommendation (.174). Team commitment to the simulation was measured by the number of group meetings herd per week and the use of decision support tools to improve decision-making. Students were asked if they used spreadsheets, word processing, and graphics software as a part of their decision making process. The use of these decision aids has grown dramatically in recent years. Earlier studies by Wolfe (1976) and Philippatos and Moscato (1969, 1971) indicated that students do not use decision-making aids or analyze background or historical information to a significant degree. However, a 1988 study by Keys et. al. found that students who used a decision support system had more positive overall attitudes toward game play than students that did not use a DSS. Our study showed that 69% of the students used spreadsheets, 71% used word processing, and 67% used graphics software to aid decision making. The correlation coefficients for these measures of team commitment are shown in Table 4. All of the variables measuring team effort appeared to have a significant impact on perceived satisfaction with the game. This is interesting in light of the earlier finding that individual commitment (as measured by number of hours spent on the game per week) did not seem to affect perceived benefits. The final statistical analysis performed was to build a regression equation, which integrates the previously examined variables into a multivariate model of student perceived learning. Regressions were run for both perceived usefulness (USEFUL) and strength of recommendation (RECOMMEND). The results of this analysis appear in Table 5. The results in Table 5 are consistent with the correlations reported earlier, although several new variables become significant in a multivariate format. The two most important determinants of student satisfaction are company performance (PERFORM) and simulation weighting in determining course grade (WEIGHT). The use of decision support tools was significant for both equations in Table 5, although the use of spreadsheets proved more significant for determining perceived usefulness, while word processing and graphics were significant in the equation for strength of recommendation. The simple correlations for economic variables were not significantly related to simulation ratings, but became significant in the equation for strength of recommendation. The negative sign on economic volatility would indicate that the uncertainty created by operating a company in a turbulent economy leads to student dissatisfaction with the game. This does not support Cohen and Rehnman’s (1966) conclusion that increasing the dynamics of the game situation should result in increased excitement and enjoyment of play. Similarly, strong growth in the economy leads to better operating results for all participants and consequently higher simulation ratings. Surprisingly, the number of worlds per class (WORLDS) entered the equation for perceived usefulness with a positive sign. This would indicated that students find games relatively more worthwhile in classes where more than one industry is being run simultaneously. CONCLUSIONS Student perceptions of simulation usefulness can be significantly linked to a variety of factors in a business game. We found that determinants of perceived learning can be grouped into four areas: course structure, parameters of the game, student characteristics, and game effort and performance. Of these categories, the first two can be directly manipulated by an instructor. Our study indicates that students perceive greater simulation benefits if the simulation is explicitly integrated into course content and if a relatively high weight is placed on the simulation in determining final course grades. We found that simulations were generally well received and were rated about equal to other teaching approaches. By manipulating game parameters instructors can also increased perceived learning. We Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 95 found that smaller team sizes and strong but stable economies led to the highest student ratings. Although student characteristics, performance, and commitment are harder to influence, we found they had a significant bearing on simulation ratings. Students who met more frequently with their groups and who used decision support tools appeared to find the simulation more useful. Students who expected to do well in the course also rated the simulation more highly, but this relationship has questionable causality. We also found that job responsibilities in the simulation affected game rating, although this finding may be unique to the specific simulation under investigation. Perceived learning and actual learning are not the same thing. Other authors have attempted to relate simulation characteristics to actual learning with mixed success. Assessing and controlling for actual learning present substantial empirical difficulties. This study relied on measures of perceived learning as a proxy for simulation effectiveness. Although it is an imperfect measure of effectiveness, at the very least it provides an index of student interest and motivation. Simulations which students find motivating and interesting are more likely to enhance commitment and learning from the experience. We have found that the context in which a simulation is used can have a direct affect on how students rate the use of a game. REFERENCES Anderson, L.F. et al A comparison of simulations case studies, and problem papers in teaching decision-making (Cooperative Research Project No. 1568, Office of Education, U.S. Department of Health, Education, and Welfare). Evanston, IL: Northwestern University, 1964. Anderson, P.H., & Woodhouse, RH The perceived relationship between pedagogies and attaining objectives in the business policy course. Proceedings of the Eleventh Annual Conference of the Association for Business Simulation and Experiential Learning, D.M. Currie & J.W. Gentry (eds.), 1984 Boseman F.G., & Schellenberger, R.E. Business gaming: an empirical appraisal. Simulation & Games, 1974, 5, 383-402. Brenenstuhl, D.C. Cognitive versus affective gains in computer simulations Simulation Games, 1975, 6, 303-311. Cohen, K.J., & Rhenman, E. The role of management games in education and research. Management Science, 1961, 7, 131-166. Dill, W.R. et al. Experiences with a complex management Game. California Management Review. 1961, 3, 38-51. Dill, W.R., & Doppelt, N. The acquisition of experience in a complex management game. Management Science, 1963, 12, 30-46. Faria, A.J. A survey of the use of business games in academia and business. Simulation & Games 1987, 18, 207- 224. Greenlaw, P.S., & Wyman, F.P. The teaching effectiveness of games in collegiate business courses. Simulation 7 Games, 1973, 4, 259-294. Kaufman, FL. An empirical study of the usefulness of a computer-based business game. Journal of Educational Data Processing, 1976, 13, 13-22. Keys, B. Total enterprise business games. Simulation & Games, 1987, 19, 225-241. Keys, B., Burns O.M., Case, T., & Wells, R.A Decision support package in a business game: performance and attitudinal effects. Simulation & Games, 1988, 19, 440-452. Miles, W.G. Jr., Biggs, W.D., & Schubert, J.N. Student perceptions of skill acquisition through cases and a general management simulation Simulation & Games, 1986, fl, 7-24. Porter, L.W., & Lawler, E.E. Managerial attitudes and performance Homewood, IL: Irwin, 1968. Philippatos, CC. & Moscato DR. Experimental learning aspects of business game playing with incomplete information about the rules. Psychological Reports, 1969, 25, 179-486. Philippatos, G.C. & Moscato, D.R. Effects of constrained information on player decisions in experimental business simulation: some empirical evidence. Journal of the Association for Computing Machinery, 1971, 18, 94-104. Raia, A.P A study of the educational value of management games. Journal of Business, 1966, 39, 339-352. Remus, W. Who likes business games? Simulation &Games, 1977, 64.68. Remus, W., & Jenner, S. Playing business games: expectations and realities. Simulation & Games, 1981, 12, 480-488. Roberts, R.M, & Field, S.E. Using student opinions in evaluating results with a business game. Proceedings of the Second Conference of the Association for Business Simulation and Experiential Learning, R.H.Buskirk (ed.), 1975 Rowland, K.M., & Gardner, D.M The uses of business gaining in education and laboratory research. Decision Sciences, 1.973, 4, 268-283. Strother, GB, Johnson, A.C., & Thompson, H.E. Educational applications of management Games: final report. Washington, D.C.: Office of Education, Department of Health Education and Welfare, 1966. Sugges, P.R. Jr. How to use business games in the business policy course: the students’ perspective. Proceedings of the Tenth Annual Conference of the Association for business Simulation and Experiential Learning, L.A. Graf & D.M. Currie (eds.), 1982. Wolfe, D.E. & Byrne E.T A comparison of perceived learning in three pedagogically different sections of a required business policy course. Proceedings of the Third Annual Conference of the Association for Business Simulation and Experiential Learning, B.H. Sord (ed), 1976. Wolfe, J. Effective performance behaviors in a simulated policy and decision-making environment Management Science, 1975, 872-882. Wolfe, J. The effects and effectiveness of simulations in business policy teaching applications. Academy of Management Review, 1976, 2, 47-56. Wolfe, J. An evaluation of a simulation-based business school curriculum integration effort. Journal of Business, 1977, 50, 343-355. Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 96 Wolfe, J. The teaching effectiveness of games in collegiate business courses: a 1973.1983 update. Simulation & Games, 1985, 16, 251-288. Wolfe, J., & Guth, G.R. The case approach versus gaming in the teaching of business policy: an experimental evaluation. Journal, of Business. 1975, 48, 349-364. Table of Contents Volume 17, 1990 The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - Revisited The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise Use and Effectiveness of an Analogy-Based Expert System Suggestions for Computerized Business Authors Dealing with Power: An Experiential Exercise Using Movie and Personal Diary Analysis Techniques A Model for Developing Student Skills and Assessing Outcomes Through Outdoor Training Computer-Aided Exercises Versus Workbook Exercises as Learning Facilitator in the Principles of Marketing Course An Exposition of Guilford's Si Model as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education Formal Planning and Simulation Team Performance: A Cross Sectional Approach Cases: Real Organizations in Real Time in the Classroom An Empirical Investigation of the Internal Validity of Marketing Simulation Game An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Game in a Principles of Marketing Course Factors Affecting Effective Teaching of Strategic Planning: Some Preliminary Evidence An Experiential Exercise for Learning About the Relationship Between Organizational Form & the Project Management Process Accounting Communication Skills can be Taught in the Auditing Course Modeling Cost Functions in Computerized Business Simulation: An Application of Duality Theory and Sheppard's Lemma A Life Cycle Analysis of Decision Making for a Strategic Management Team What's the Problem? A Dynamic Model for Teaching Problem Solving Skills Experientially International Currency Fluctuations: Money$im, A Simulation Superstores: A Specialized Retailing Simulation Within a Specialized Marketing Curriculum Factors Affecting Student Perceptions of Learning in a Business Policy Game VC + EL = VL The Name Game: An Experiential Exercise in Intergroup Relations The Effects of Experiential Accounting Work Experience on Student Performance in Intermediate Accounting Courses The Results of Using the Experiential Activity Group Performance Evaluation in a Business Policy Setting Using a Legal Database to Describe the Legal Environment of Marketing (and Business) Matching Environmental Uncertainty and Organizational Configuration An Instructional Computer Simulation of Tampering in QC Executive Evaluation of Student Learning in the Looking Glass Simulation Group Personality Composition and Total Enterprise Simulation Performance An Expert System for Selecting Analytical Techniques for Analyzing Marketing Research Data Effects of Cognitive Styles on Responses in an In-Basket Simulation A Psychometric Analysis of Kolb's Revised Learning Style-Inventory Selecting and Developing Experiential Exercises Using Movies Application of a Real-World Strategic Management Model in the Classroom Demand Equations which Include Product Attributes Consumption as the Objective in Computer-Scored Total Enterprise Simulations The Effects of Decision Format and Evaluation on Simulation Performance, Decision Time, and Team Cohesion The Effects of Computer Related Assignments on Student Performance in Business Administration The Money Game: A Dynamic Simulation Including Random Shocks for Money and Banking Courses Methods for Evaluating Performance on Business Simulations: A Survey The Effects of Synergogy on the Policy Course: Significant Improvements in Student Learning and Teacher Evaluation Conditions and Outcomes of Trust in a Two-Person Bargaining Exercise Bankgame Enhancing Computer Business Simulation with the Use of VGA Graphics An Experiential Approach to Entrepreneurship An Advanced Simulation Method (ASM) for Multiple Objective Problems The Influence of Experiential learning Techniques on Student Recognition of Non-Primary Learning Styles Negotiating Mergers and Acquisitions: A Cocktail Napkin Approach Identification of Unintended Effects in Experiential Laboratory Exercises An Experimental Comparison of Paper and Pencil and Computer Aided Decision Support Tools Porting a Simulation from the IBM World to the Macintosh World A Transaction Cost Analysis of Experiential Learning The Development of Experiential Exercises for Courses in Entrepreneurship and Small Business Management The Assessment Center as and Experiential Classroom Exercise Pricing Strategy Algorithms for Playing Business Simulations Organizational Structures for International Operations: An Experiential Activity Simulation Emphasis in the Business School Capstone Course An Integrated Approach to Computerizing the Business Curriculum Cross-Cultural Business Negotiations Exercise Organizational Socialization and Gender Differences in Students at Work Understanding Student Work Experience: A Content-Analytic Approach Introducing Executive MBA Programs with Management Games An Analysis of Improvement in Business Decision Outcome with Sequential Use of Two Simulation Games Teaching Business Policy Utilizing Mass Lecture and Individual Case Labs Potholes Along the Road to Evaluating Learning Outcomes: The Case of Outdoor Management Training Experiential Learning for Interior Design Students: Using CADD, Lotus 1-2-3, and Wordperfect Cognitive Learning Using a Computer-Based, Qualitative Interactive Business Simulation Sex Discrimination: Does the Woman get the Job or Does the Best Man Win? A Search for Visual Aids to Support Experiential Learning Through the 1990's An Experimental Analysis fo the Effectiveness of Student Role-Playing in Sales Training How to Have Students Learn from their Term Projects Self-Evaluation Exercise (SEE): An Assessment of Class Contribution A Study of the Influence of Team Formation on Attitudes and Performance in Management Games A Hardware Based CIM Simulation Laboratory Model Test Substance Abuse in Organizations Micro Computer Training Models Teaching Forecasting, Cash Budgeting and Inventory Model Building Using SBTools A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes An Adjunct Writing Instruction Assistant, The Computer; With an Illustration Classroom Software for ABC Analysis Utilizing Information Processing Technology to Enhance the Business Policy Simulation Experience Progressive Cases Realistic Job Previews Vs. Traditional Job Previews: Experiencing the Differences and Understanding the Consequences Investment Analysis Using the Pragmatic Multiplier Approach A Computer Simulation Interface for Competitive and Firm Analysis Self-Assessment of Ethical Decision Making Predispositions Preparing Managers for Overseas Assignments An Inquiry into Japanese Marketing: Workshop on Teaching Japanese Marketing The Performance Appraisal Feedback Interview: A Role Play for Human Resources Management Teaching Counselor Selling techniques Using Experiential Techniques to Teach International Topics International Management Simulation Gaming: Current Status and Future Developments A Realism Comparison of Simulation Technologies/Methodologies A Time-Efficient Game to Illustrate Concepts Taught in Management Courses and Management Development Programs Teaching the Management of Technology The Concept of Face and the Applicability of Experiential Exercises in an Oriental Culture's