PERCEPTIONS OF LEARNING IN TE SIMULATIONS Developments in Business Simulation & Experiential Learning, Volume 26, 1999 PERCEPTIONS OF LEARNING IN TE SIMULATIONS Jerry Gosen and John Washbush University of Wisconsin-Whitewater ABSTRACT The purpose of this study was to explore the construct perceived learning and its relationship to objective indicators of learning and also to performance. In this study, perceptions of learning were divided into ten types for measurement purposes. The study was undertaken with seniors playing a total enterprise simulation. The results showed no relationship between perceptions and objective measures of learning. In other words, those who perceived that they learned a lot did not attain either high or low objective learning scores. On the other hand, there was a positive relationship between perception of learning and performance. INTRODUCTION In an extensive review of research on learning effectiveness, Anderson and Lawton (1997) clearly distinguished between perceived and objective effects. Perceived effects are from studies where participant perceptions are used as the dependent variable for a learning outcome, and objective effects are employed if a form of objective measure was used as the dependent variable for a learning outcome. Others, including Gentry et at. (1997), Lawton and Anderson (in press), Parasuraman (1980), and Parasuraman (1981), distinguished between perceived and objective measures of learning. These authors considered perceived learning to be less valid or suitable than objectively measured learning. For example, Lawton and Anderson (in press) stated that perceptions and attitudes have been overused because we know how to use them, and we will not succeed in assessing the educational merits of simulations until we develop better, more objective, dependent measures. Gentry et al. (1998) placed perceptions of learning into a ‘feel good’ category of measures. According to these authors positive perceptions of learning are a part of a halo effect that stems from that fact that students enjoy the experience. In summary, perceived learning has been judged inferior because it is subjective and because it is easier to measure. However, we don’t know very much about perceived learning. Also, we don’t know if there is any relationship between perceived and other (more objective) measures of learning. This study focused on perceived learning. It dealt with the relationship between perceived learning and objective learning and also with the relationship between perceived learning and simulation performance. Specifically this study asked four questions: 1. What do students think they learn from a simulation? 2. Do those who learn the most, measured objectively, perceive that they learn a great deal? 3. What do those who score the highest on objective learning measures believe they learn? 4. Do those that perceive that they learn a tot also perform well? METHOD Subjects and Procedure The subjects of this study were 136 students enrolled in four sections of the required un 170 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 dergraduate Administrative Policy course at the University of Wisconsin-Whitewater during the Fall 1997 and the Spring 1998. There were four industries with 40 three-member teams, 4 two- member teams, and 2 four-member teams. The simulation length varied between 12 and 16 quarters. The Simulation used was MICROMATIC (Scott et al., 1992). Variables and Variable Measurement Objectively Measured Learning. We assessed objectively measured learning with two forms of a multiple-choice and short-essay examination. These forms were made deliberately parallel in form and content. The examinations were constructed using questions and situations routinely confronted by companies competing in MICROMATIC. These include manipulating and analyzing the marketing-mix, making operating decisions, determining costs of goods sold, and understanding the consequences of doing or not doing ratio analysis or cash flows. Many of the questions require analysis, calculations, and the application of principles from MICROMATIC. For all industries, Form 1 was administered as a pretest at the beginning of the semester. Form 2 was administered at the end of the semester. Learning over the period of play was defined as the percentage score for Form 2 minus the percentage score for Form 1. The test developers used a common scoring key for all questions to ensure uniformity of measurement. Statistical reliability estimates for the instruments have ranged from .65 to .7. Perceptions of Learning. Perceptions of learning were measured with a ten-item Likert style questionnaire, asking players to state their beliefs about the degree to which they learned each of the ten kinds of learning. The ten items represented the most frequently given answers to the question, “What have you learned while playing the game,” which was given to simulation players in the Falls of 1995 (Gosen and Washbush, 1997) and 1996 (Gosen and Washbush, 1998). Students responded to the questionnaire during one of the last two quarters of simulation play. The ten items are stated in table 1 along with mean responses to the question, “To what degree did you learn (each of the perception items).” Informally, the senior author categorized the items of the pre- and post-tests according to the ten perceptions of learning items and restated these ten as learning goals. For example the perception that one learned the cause and effect principles of the game was restated ‘to learn the cause and effect principles of the game.’ The result of this categorization was that of the 37 items of the pre-test and the 35 items of the post test, the majority tapped two goals: ‘to learn the game’s cause and effect principles’ and ‘to anticipate and plan for future events.’ Performance. Performance scores were generated by the game’s scoring routine. They were based on Net Income (40%), Return on Sales (30%) and Return on Assets (30%). Game performance was worth 15% of the course grade, 5% of the course grade was based on peer ratings of team contribution, and 5% of the course grade reflected the score on the post test exam measuring learning in the simulation. 171 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 Table 1: The ten perception items and the average response to the “To what degree did you learn” question Fall 1997 Spring 1998 To correct or compensate for earlier made 3.74 3.83 mistakes The game’s cause and effect principles 3.46 3.54 Forecasting skills 3.56 3.46 More about game playing, that is adapting to 3.49 3.34 rules and competitor actions That the game and business requires consid- 3.79 3.51 eration of complex phenomenon Financial statement analysis skills 3.39 3.31 To plan strategically 3.66 3.66 Ratio analysis skills 3.12 To anticipate and plan for future events 3.81 3.69 To balance numerous perspectives and priori- 3.49 3.63 ties RESULTS The results yielding the average responses to the question, “To what degree did you learn each of the ten kinds of learning” is contained in table 1. In general students felt that they learned to correct and compensate for earlier mistakes, to plan strategically, to anticipate and plan for future events, and that the game and business requires the consideration of complex phenomena to a relatively great degree, and they learned financial and ratio analysis and about game playing (that is adapting to rules and competitor actions) to a relatively small degree. These results, which provide an answer to this study’s first question, “What do students think they learn from a simulation” make some sense. Strategic analysis and the complexity of business decisions are new phenomena for most seniors, and it make sense that they learned them as the game presented them. In contrast, many seniors have been exposed to ratio and financial analysis in earlier courses, and it makes sense that fewer students felt that they were learning these to a great degree with a capstone simulation. The correlational results of this study are contained in table 2. Of nineteen correlations between perception scores on one hand and scores on the objective measure of learning on the other, only one was positive and significant, and that correlation only explained about five percent of the potential variance between the two variables. Most of these correlations were close to zero. These results suggest little or no relationship between objective learning and perceptions of learning. Those who scored highest on the objective measure learning did not perceive that they learned any more or less than those who scored lower on the objective measure. Thus, the answer to this study’s second question, “Do those who learn the most, measured objectively, perceive that they learn a great deal?” is ‘no.’ 172 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 Table 2: Correlations of the 10 kinds of perception scores with objective learning and performance scores Correlation with Objective Correlation with Learning Score Performance Fall 97 Spring 98 Fall 97 Spring 98 To correct or compensate for earlier made mis- .072 -.077 .140 .121 takes The game’s cause and effect principles .009 -.135 .156 .151 Forecasting skills .243* .058 .407* .178 More about game playing, that is adapting to rules .071 -.375* .242* .354** and competitor actions That the game and business requires consideration -.033 -.026 .117 .286* of complex phenomenon Financial statement analysis skills .001 .107 .023 .306* To plan strategically .046 -.101 .131 .240* Ratio analysis skills .049 .461* * To anticipate and plan for future events .135 .000 .239* .107 To balance numerous perspectives and priorities .006 -.135 .234 . 285* * p less than .05 ** plessthan.01 This study’s third question concerned the kind of perceptions of those who scored well on objective measures of learning. The results show that, in the Fall of 1997, there was a slight, significant relationship between the objective measure and the degree to which students perceived or thought that they learned forecasting. In the Spring of 1998, there was a significant negative correlation between the objective measure and the degree to which students thought they learned about game playing. So one might argue that those who scored the high on the objective measure think that they learn some about forecasting, and those that scored low on the objective measure think the learn about game playing but those relationships occurred only one of two semesters. And that was all. For the most part, the results of this study show no pattern of ‘perceived learnings’ for those who score high (or low) on this study’s objective measure of learning. This study’s fourth question asked whether there was any relationship between perceptions of learning and performance in the simulation, and apparently the answer to question 4 is yes. Of the 19 correlations in table 1 between perceived learning and performance, all were positive and nine were significant. While these correlations were not extremely high, these results suggest that those that perform the best in the simulation also perceive that they learn the most. DISCUSSION These results shed some light on the concept, perceived learning, and this variable’s relationship to objective measures of learning and 173 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 also to performance. From this study’s results, perceived learning appears to be unrelated to objective indices of learning. This implies that self reports of what one learns and objective measures of what one learns somehow tap different things. Assuming that objective measures of learning are ‘true’ indicators of what one learns (a tentative assumption), then in general what one thinks he or she learns is not the same thing as what he or she actually learns. These result support the contentions of such authors as Lawton and Anderson (in press) and Gentry et al. (1998) who treat perceptions of learning and objective indices of learning as different phenomena. An inspection of both the specific perceptions of learning recategorized as learning goals and the items of the pre- and post learning tests used in this study, further supports the notion that perceived and objective indices of learning are different from each other. As indicated in the method, of the ten specific perceptions of learning dimensions, the study’s objective learning instrument tapped two to a greater degree than others. It tapped the individual’s ability to learn the game’s cause and effect principles and the individual’s ability to anticipate and plan for future events. If perceptions of learning and objective indices were tapping the same dimensions, then correlations between the objective measure and those two perceptions of learning should be higher than correlations with the other perception items. As indicated in table 1, they were not. Correlations between the objective measure and the perception that individuals were mastering the game’s cause and effect principles and anticipating and planning for future events were near zero, just as most of the other correlations between the objective measure and perceptions were near zero. On the other hand, the results do show a positive relationship between the objective measure of learning and performance. To us, these indicate that those who were performing well thought that they were learning a lot. These results appear to confirm the notion of Gentry et al. (1998) that there is a halo effect that reveals itself in perceptions of great(er) amounts of learning. It appears that in this study the halo effect came from performing well in game competition. REFERENCES Anderson, P.H. & Lawton, L. (1997). Demonstrating the learning effectiveness of simulations: where we are and where we need to go. Developments in Business Simulation & Experiential Exercises, 24, 68-73. Butler, R.J., Markulus, P.M., & Strang, D.R. (1985). Learning theory an research design: How has ABSEL fared? Developments in Business Simulation & Experiential Exercises, 12, 86-90. Gentry, J.W., Commuri, S.R., Burns, A.C., & Dickenson, J.R. (1998). The second component to experiential learning: A look back at hoe ABSEL has handled the conceptual and operational definitions of learning. Developments in Business Simulation & Experiential Exercises, 25, 62-68. Gosen, J. & Washbush, J. (1997). Antecedents of learning in the simulation. Developments in Business Simulation & Experiential Exercises, 24, 60-67. Gosen, J. & Washbush, J. (1998). Antecedents of learning in the simulation revisited. Developments in Business Simulation & Experiential Exercises, 25, 152-153. 174 Developments in Business Simulation & Experiential Learning, Volume 26, 1999 Lawson, L. & Anderson, P.H. (in press). Measuring what is learned from simulations: Future directions for research. Simulation & Gaming: An International Journal. Parasuraman, A. (1980). Evaluation of Simulation Games: A critical look at past efforts and future needs. Developments in Business Simulation & Experiential Exercises, 7, 192-194. Parasuraman, A. (1981). Assessing the worth of simulation games. Simulation and Games, 12, 189-200. Scott, T.W., Strickland, A.J., Hofmeister, D.L., & Thompson, M.D. (1992). MICROMATIC: A Management Simulation. Boston: Houghton Mifflin. 175 Table of Contents Volume 26, 1999 ABSEL's Historical Research Interests Back From the Future: An ABSEL Merlin Exercise for the Year 2005 The Contributions of ABSEL During the 1980's ABSEL's Contributions to Experiential Exercises in the 90's ABSEL's Contributions to Experiential learning/Experiential Exercises: The Decade of the 1970's Business Simulations - Algorithms and Model Enhancements: A 25 year Review A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning A Framework for Assessing the Competencies Reflected in Simulation Performance Developing A Learning Culture: Assessing Changes in Student Performance and Perception Financial Simulation Using Distributed Computing Technology Analyzing Managers' Judgements and Decisions with an Educational Business Simulation Understanding Your Business through Home-Made Simulator Development An Examination of a Reanalysis of the Impact of a Market Leader on Simulation Competitors' Strategies Applying Shocks to TE Simulations: A Demonstration Increasing Efficiency of Management Skill Assessment A Testbank for Measuring Total Enterprise Simulation Learning Developing Leadership Skills - Video Live! LEADSIMM: Collaborative Leadership Development for the Knowledge Society A Team Approach to Producing Multi-Media Laptop and Video Formatted Presentation Tinkertoys Revisited: Exploring Trust Based Relationships Overall Dominance in Total Enterprise Simulation Performance Success or Bankruptcy: The Relationship between Personal and Goal Orientation and Simulation Performance Assessing the Effects of Feedback: Muti-method and Muti-directions in Multi-pedagogical Courses The Missing Ingredients in Experiential Learning Purpose and Learning Benefits of Business Simulations: A Design and Development Perspective Building Capabilities for Change through Laboratory Simulations Modeling Innovation as a Process of Design in Educational Business Simulation The Need to Measurer variance in Experiential Learning and a New Statistic to do so Assessing Effectiveness of an Experiential Oriented Course Over Time Developing Participant Satisfaction Models of Experiential Exercises in Business Education Perceptions of Learning in TE Simulations Students' View on the Use of Business Gaming in Hong Kong Management Gaming's Lost Opportunity: Meditations about the Russian Experience and Prospects for the Future Unanticipated Enhancements in the Business Strategy and Policy Game when Running in Windows 95 Using a Business Game to Demonstrate Broad Business Concepts A New Model for Business Courses (Getting the Student Connected) Seeing the Forest and the Trees: Integrating Knowledge Using Large Scale Simulations in Capstone Business Strategy Classes Is It Here To Stay? A Roundtable Discussion of the Inter-Group Interaction Interactive Tools Used in Applying Financial Concepts Strengthening Essential Skills through a Finance Exercise: Calculation of Beta A Model of Currency Exchange Rates Understanding Currency Exchange Rates: A three-part Exercise Student Experiences in the World Intercollegiate Business Game Competition Student Experiences in the International-Collegiate Business Policy Game Competition Using Boards of Directors in Simulation Environments: Comments From Board Members Sharing Best Practices: Teaching Smarter, Not Harder Star Power: A Simulation for Understanding Power and Empowerment The Marketing Game: A Marketing Principles Simulation Alexander Islands: GSSM Tiny Business Simulator on the WWW Putting Strategy into Strategic Business Games Industry Analysis, Porter's Five Forces Model and Strategic Group Maps in the Business Strategy Game Simulation The Use of Concept Mapping to Improve Student Performance and Understanding of Strategic Management Concepts: A Comparison of Techniques Transformational thinking in the Organizational Behavior Course: The Use of Metaphor as an Assessment Tool Creation of a Virtual Learning Community for the Global Virtual Enterprise Project Inter Institutional use of Educational Resources: Joint Use of Management Simulation Games Business Insights: Theory and Practice with the Aid of a Business Simulation Progress: An Experiential Exercise in Development Marketing The ABC's of teaching the Theory of Constraints to Undergraduate Business Students Demonstrating Principles of Organizational Purchasing Behavior through an Experiential Exercise The Virtual Manager: A Different Simulation for Managing Complexity When the Rules are changing and Chaos Breeds Innovation: Recapturing the Value of Constructive Thinking and Play in Simulation Training The Pitfalls, and Potential, of Actual Events as Problem Drivers Using Business Games to teach Environmental Awareness and Green Management: The International Experience with the ENSIM Game A Review of my ABSEL-Related Work Simulation of Government: A Workshop Using GEO Creating Internet-Based Games Using Perl and JavaScript Who's on First? Exploring the Concepts of Problem-Based Learning, Experiential Learning, and Lifelong Learning Students Learn Customer Service and Selling while Conducting Research So You Want to Run an NFL Football Team–An Honors Interdisciplinary Project Supervised Internship: The Employer's Perspective Using Computer Assisted Simulation to Teach International Business Strategy: A Case of the Multinational Management Game (MMG) Multi-Cultural Experiential Learning: A Computer Simulation in China An Appreciative Stance on Diversity as We Move into the 21st Century: A Timeline Exercise to Identify Key Experiences in Good Work Relationships between Black and White Peers A Day in the Life of an Interactive, Real Time, and Internet Delivered Course: A Demonstration Comparing Internet Search Engines: An Experiential Learning Exercise Total Enterprise Simulations and the Internet: Assessing Student Perceptions and Preferences Teaching Accounting Information Systems in a Practicum Format Providing Experiential Learning in Accounting through a Field Study Payroll Project A Spreadsheet Based Business Simulation Game Computer-Behavioral Simulations Training for Project Managers Simulation Scenarios - Rationale and Illustration