A FRAMEWORK FOR ASSESSING THE COMPETENCIES REFLECTIED IN SIMULATION PERFORMANCE Developments in Business Simulation and Experiential Learning, Volume 26, 1999 A FRAMEWORK FOR ASSESSING THE COMPETENCIES REFLECTED IN SIMULATION PERFORMANCE Hugh M. Cannon, Wayne State University Alvin C. Burns, Louisiana State University ABSTRACT This paper is part of an on-going stream of re- search designed to determine the potential of using simulation games as instruments of as- sessment. Previous research has failed to vali- date games for this purpose, in part, because it has not specified what competencies the games should assess. This paper seeks to address the problem by proposing a three-dimensional tax- onomy of performance competence. It then dis- cusses how researchers might operationalize the three dimensions, thus developing alternative competency measures to which game perform- ance might be compared to establish its valid- ity. INTRODUCTION In 1990, Keys and Wolfe suggested that "man- agement games will play a more significant role in management development and assessment ef- forts in business schools as part of the move to- ward competency-based outcome measure-ment" (p. 324). This provided an impetus for The Asso- ciation for Business Simulation and Experiential Learning (ABSEL) to organize a Committee on Assessment (Thavikulwat, 1995) to pursue the use of games as measures of measures of mana- gerial competence. The Committee drafted stan- dards and registration procedure for assessment instruments (Cannon, Frazer, Goff, Markovich, Stevens, and Thavikulwat (1996), presented them at ABSEL's 23rd Annual Meeting in Or- lando, Florida, and published the final version in ABSEL News & Views ("Standards and Registra- tion Procedure," 1996). The standards applied to both simulation games and/or experiential exer- cises. While several games were submitted for registra- tion, the Committee found itself frustrated by a lack of evidence that games truly measured rele- vant managerial competence. Evidence would take the following forms: 1. Reliability in the instrument's measurements; 2. Discrimination by the instrument between in- dividuals within a population with different types and/or degrees of learning; 3. Convergence between the instrument's measure and other reasonable measures of learning; 4. Normative scores for different populations. In a paper presented to the 1998 annual ABSEL conference, Anderson, Cannon, Malik and Thavikulwat (1998) addressed the problem of validation from two perspectives. First, the tradi- tional gaming literature has sought to determin- ing the instructional validity of simulations by showing how they contribute to reliable and valid educational outcomes (Burns, Gentry, & Wolfe, 1990; Keys & Wolfe, 1990; Stanislaw, 1986). This is illustrated by the left-hand side of Exhibit 1. Second, the movement toward using games as tools of assessment seeks to determine validity of game performance as an indicator of managerial competence. EXHIBIT 1 SIMULATION GAMES AS ASSESSMENT INSTRUMENTS VS. EDUCATIONAL TOOLS Educational Educational Educational O s 40 Simulation games as a method of experiential learning Process Outcomes Simulation game performance as a method of educational assessment Assessment Validity as a method of teaching Validity as an assessment instrument Performance constructs representing key business skills bviously, the two perspectives are related. They hare a common set of performance constructs Developments in Business Simulation and Experiential Learning, Volume 26, 1999 that provides both the measure of educational success and of managerial competence. But this leads to a potential problem. But what if success in the simulation game does not represent mana- gerial competence, but rather, some other con- struct – such as ability and motivation to play simulation games? If this is the case, teachers who use simulations as educational tools will naturally structure their teaching to promote game performance. Students who respond well to teaching activities will tend to do well and appear to be effective managers. This reinforces the no- tion that simulation performance is a valid meas- ure of competence, even when it is not. The real question, then, addresses the nature of simulation performance. Does it really represent managerial competence, competence as a stu- dent, or some other construct all together? The only way to answer the question is by studying the nature of simulation performance, comparing it to other measures of competence to see if it ap- pears to be related. The purpose of this paper will be to take another step in this line of research. Performance compe- tence is not an easy thing to measure. Research suggests that it involves a number of different dimensions. Our job is to develop a theoretical framework which not only helps us understand the nature of performance competence, but which leads us closer to our ultimate goals of re- liable and valid measurement. TOWARD A FRAMEWORK FOR EVALUATING PERFORMANCE COMPETENCE The quest for a valid set of performance con- structs is by no means new to education. In the late 1940s, a committee of renowned educational psychologists sought to develop a comprehensive set of educational objectives. The result was a six-level taxonomy that suggested performance involved six increasingly demanding levels (Bloom, Englehart, Furst, Hill, and Krathwohl 1956). These ranged from simple knowledge to the ability to create new ideas and evaluate them according to their merits. They have come to be known commonly today as Bloom’s Taxonomy of educational objectives. Note that Bloom’s taxonomy addressed cognitive learning, or learning that addresses how to con- sciously acquire and manipulate ideas. Sub- sequent work addressed the question of feelings – values and attitudes. It acknowledged the fact that success is more than thinking. It involves a great deal of socialization, molding to the culture in which business (or other types of) success takes place. The result was a second, affective, taxonomy (Krathwohl, Bloom, and Masia 1964) which involved such things as the propensity to pay attention to the appropriate cues, to the ten- dency to prioritize effectively, to the internali- zation of a set of appropriate values as a basis for governing one’s behavior. Finally, the third dimension addressed the fact that neither conscious knowledge nor values and attitudes are sufficient to explain effective performance as we observe, or indeed, as we experience it. There is a third dimension – the ability to act quickly and effectively on an al- most unconscious, instinctual level. This kind of ability, or skill, is incorporated in a psycho- motor taxonomy (Simpson 1974). It addresses such things as the ability to carry out a specific sequence of guided activities to the ability to improvise appropriate sets of complex behav- ior. Exhibit 2 summarizes the three dimensions of learning. EXHIBIT 2 HIERARCHICAL LEVELS THE THREE DIMENSIONS OF PERFORMANCE Cognitive Domain Knowledge, or the ability to recall ideas such as facts, concepts and theories Comprehension, or the ability to understand and make intellectual use of knowledge. Application, or the ability to use abstract ideas in concrete situations. Analysis, or the ability to break ideas down into their parts and logical premises. Synthesis, or the ability to develop new ideas 41 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 from apparently unrelated parts. Evaluation, or the ability to judge the merit of ideas for given purposes. Affective Domain Receiving, or the tendency to recognize and pay attention to important stimuli. Responding, or the tendency to act in appro- priate ways as a result of a stimulus. Valuing, or the internalization of underlying motives that govern response. Organization, or the arrangement of values into a coherent, stable system. Characterization by a value, or the use of values to control one’s behavior. Psychomotor Domain Perception, or the ability to sense objects, qualities and relationships via sensory organs. Guided response, or the ability to perform a specific act under the guidance of a teacher. Mechanism, or the ability to perform a learned act habitually without guidance. Complex overt response, or the ability to per- form a complex pattern of acts. Adaptation, or the ability to alter an act to meet the demands of anew situation. Origination, or the ability to develop new acts through the application of unrelated skills. These three hierarchies provide a useful starting place for developing performance criteria. While they are too abstract for direct applica- tion, they can serve as a template for identify- ing the types of performance through which simulation games might be evaluated. Exhibit 3 illustrates the process. The Exhibit suggests that the process of devel- oping assessment criteria should proceed in parallel, building from both game performance and actual job performance. That is, evaluators would consider both the design of games and the design of the organizations the games are intended to simulate (boxes 1 and 6). In each case, the results of the analysis would be fil- tered through the lens of educational objectives (box 2). For instance, an evaluator might iden- tify performance behaviors and requirements involved in sales forecasting by asking how forecasting relates to a set of questions growing out of Exhibit 2. What knowledge do students or workers need? What do they need to com- prehend? To what extent do they have to apply general forecasting models to the specific re- quirements of a particular forecasting problem? To what extent must they analyze potential in- formation sources to extract relevant data? Must they synthesize of potential information sources and forecasting principles to develop better approaches? To what extent must they evaluate how well the approaches are likely to work? EXHIBIT 3 A MODEL FOR DEVELOPING PERFORMANCE ASSESSMENT CRITERIA H ierarchies of Educational O bjectives Perform ance scale developm ent Game design and performance Organization design and performance Specific game performance behaviors/ requirements Specific organizational performance behavior/ requirements General organizational performance evaluation General game performance evaluation Convergent validation 1 3 2 4 5 6 7 8 9 In similar fashion, how do the questions rising out of the Affective and Psychomotor domains translate into performance behaviors and re- quirements? What type of stimuli must they learn to notice? What kind of value system must they ultimately develop and incorporate into their application of the forecasting proc- ess? What kinds of patterns must they be able to recognize in the data? What kinds of com- plex behaviors must they learn to perform in order to carry out the forecasting process? To what extent must they develop new ones? The result of this process is a pool of perform- ance items that represents the ability to carry out an effective sales forecast, and by exten- sion, any other skill or activity the simulation is designed to teach and evaluate. Given a pool of 42 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 items, researchers can use standard techniques of psychometric scale development to develop multi-item performance indices (box 4). Given these indices as tools, researchers can evaluate targeted performance in both a simula- tion and actual organizational performance en- vironment (boxes 5 and 8). They can then com- pare these to establish a measure of convergent validity (box 9). CASTING THE PROCESS IN AN ON-GOING PROGRAM OF RESEARCH Note that the process described in Exhibit 3 represents a research program of enormous proportions. A single aspect of the game play- ing process – forecasting, for instance – can constitute a major research study. It involves the development of a host of items, several rep- resenting each of the various aspects of the forecasting process. And how many skills are there along the order of forecasting? EXHIBIT 4 THE THREE FACES OF PERFORMANCE-ITEM DEVELOPMENT Ultimately, the research program must proceed not only from a taxonomy of educational objec- tives, but also from an overall taxonomy of the behaviors and requirements that constitute ef- fective management in the kind of organization the simulation is trying to address. These, in turn, should all be represented by multiple items to ensure scale reliability, as suggested in Exhibit 4. One of the advantages in pursuing this kind of research in a simulation game environment is that gamers ply their trade by identifying the critical elements of organizational performance that need to be simulated. In many ways, the ul- timate test of one’s understanding of manage- ment processes is one’s ability to simulate them. SUMMARY AND CONCLUSIONS This paper is an outgrowth of an assignment made by ABSEL’s Committee on Assessment during the 1998 annual conference in Hawaii. Its purpose is to extend ABSEL’s program of research regarding the use of simulation games as instruments of assessment. While the pro- gram has been more problematic than originally anticipated, the research has continued. The Committee has retrenched from a premature call for immediate registration of games to a more systematic, theory-based approach (Anderson, Cannon, Malik and Thavikulwat 1998). Educational Objectives Mult ipl e I tem s M an ag em en t S ki lls This paper addresses the seemingly overwhelm- ing problem of identifying measurable criteria by which gaming experience can be evaluated against the demands of actual management situations. Even so, the task is daunting. It promises to involve an enormous amount of painstaking research. Nevertheless, by develop- ing a specific, theory-based program, the re- search can progress at a reasonable pace. The program itself would proceed as follows: 1.The process would begin for gamers, as it would for management system analysts, with the identification of the specific behaviors and requirements that constitute performance success. 2.In order to structure this search, and anchor it in an underlying theoretical framework, gam- ers and system analysts would view their be- 43 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 haviors and requirements through the lens of established taxonomies of educational objec- tives. Consistent with the broader view of education that characterizes the ABSEL ap- proach, these taxonomies would include af- fective and p sychomotor as well as cognitive objectives. 3.The output of the search and conceptualiza- tion process would be a set of performance items which will provide the basis for devel- oping specific performance scales. These would be developed by using standard psy- chometric scale development techniques. 4.Once developed and validated, the perform- ance scales provide the missing link in the game validation process. They provide a common instrument for comparing perform- ance in actual versus simulated decision envi- ronments. This paper provides the basic framework. How- ever, the real work is yet to come. We antici- pate that the next step in the research program will be to begin developing specific validation studies. These would provide actual experience with the practical problems researchers are likely to encounter in game validation. REFERENCES Anderson, P., Cannon H. M., Malik D. & Pre- cha Thavikulwat (1998). Games as Instru- ments of Assessment: A Framework for Evaluation. Developments in Business Simu- lation and Experiential Learning, 25, 31-39. Bloom, B. S., N. D. Englehart, E. J. Furst, W. H. Hill, & D. R. Krathwohl (1956). Taxon- omy of Educational Objectives -- The Classi- fication of Educational Goals, Handbook I: Cognitive Domain (New York: David McKay Company). Burns, A. C., Gentry J. W., and Wolfe J. (1990). A Cornucopia of Considerations in Evaluating the Effectiveness of Experiential Pedagogies, in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 253-278. Cannon, H. M., J. R. Frazer, J. Goff, D. R. Markovich, R. Stevens, & P. Thavikulwat (1996). Draft standards and registration proce- dure for assessment instruments. Developments in Business Simulation & Experiential Exer- cises, 23, 94. Keys, B. & Wolfe J. (1990). The Role of Man- agement Games and Simulations in Education and Research. Journal of Management, 16:1 311. Krathwohl, D. R., Bloom B. S., & Masia B. B. (1964). Taxonomy of Educational Objectives -- The Classification of Educational Goals, Handbook II: Affective Domain (New York: David McKay Company). Simpson, E. J. (1974)). The Classification of Educational Objectives in the Psychomotor Domain, in Robert J. Kibler, Donald J. Cegala, Larry L. Barker, and David T. Miles. Objectives for Instruction and Evaluation (Boston: Allyn and Bacon) 107-112. Standards and registration procedure for assess- ment instruments. (1996). ABSEL News & Views, 16 (Spring/Summer), 7. Stanislaw, H. (1986). Tests of computer simula- tion validity. Simulation & Games, 17, 173- 191. Thavikulwat, P. (1995). Report from the program chair. ABSEL News & Views, 15 (Spring/Summer), 1. 44 Table of Contents Volume 26, 1999 ABSEL's Historical Research Interests Back From the Future: An ABSEL Merlin Exercise for the Year 2005 The Contributions of ABSEL During the 1980's ABSEL's Contributions to Experiential Exercises in the 90's ABSEL's Contributions to Experiential learning/Experiential Exercises: The Decade of the 1970's Business Simulations - Algorithms and Model Enhancements: A 25 year Review A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning A Framework for Assessing the Competencies Reflected in Simulation Performance Developing A Learning Culture: Assessing Changes in Student Performance and Perception Financial Simulation Using Distributed Computing Technology Analyzing Managers' Judgements and Decisions with an Educational Business Simulation Understanding Your Business through Home-Made Simulator Development An Examination of a Reanalysis of the Impact of a Market Leader on Simulation Competitors' Strategies Applying Shocks to TE Simulations: A Demonstration Increasing Efficiency of Management Skill Assessment A Testbank for Measuring Total Enterprise Simulation Learning Developing Leadership Skills - Video Live! LEADSIMM: Collaborative Leadership Development for the Knowledge Society A Team Approach to Producing Multi-Media Laptop and Video Formatted Presentation Tinkertoys Revisited: Exploring Trust Based Relationships Overall Dominance in Total Enterprise Simulation Performance Success or Bankruptcy: The Relationship between Personal and Goal Orientation and Simulation Performance Assessing the Effects of Feedback: Muti-method and Muti-directions in Multi-pedagogical Courses The Missing Ingredients in Experiential Learning Purpose and Learning Benefits of Business Simulations: A Design and Development Perspective Building Capabilities for Change through Laboratory Simulations Modeling Innovation as a Process of Design in Educational Business Simulation The Need to Measurer variance in Experiential Learning and a New Statistic to do so Assessing Effectiveness of an Experiential Oriented Course Over Time Developing Participant Satisfaction Models of Experiential Exercises in Business Education Perceptions of Learning in TE Simulations Students' View on the Use of Business Gaming in Hong Kong Management Gaming's Lost Opportunity: Meditations about the Russian Experience and Prospects for the Future Unanticipated Enhancements in the Business Strategy and Policy Game when Running in Windows 95 Using a Business Game to Demonstrate Broad Business Concepts A New Model for Business Courses (Getting the Student Connected) Seeing the Forest and the Trees: Integrating Knowledge Using Large Scale Simulations in Capstone Business Strategy Classes Is It Here To Stay? A Roundtable Discussion of the Inter-Group Interaction Interactive Tools Used in Applying Financial Concepts Strengthening Essential Skills through a Finance Exercise: Calculation of Beta A Model of Currency Exchange Rates Understanding Currency Exchange Rates: A three-part Exercise Student Experiences in the World Intercollegiate Business Game Competition Student Experiences in the International-Collegiate Business Policy Game Competition Using Boards of Directors in Simulation Environments: Comments From Board Members Sharing Best Practices: Teaching Smarter, Not Harder Star Power: A Simulation for Understanding Power and Empowerment The Marketing Game: A Marketing Principles Simulation Alexander Islands: GSSM Tiny Business Simulator on the WWW Putting Strategy into Strategic Business Games Industry Analysis, Porter's Five Forces Model and Strategic Group Maps in the Business Strategy Game Simulation The Use of Concept Mapping to Improve Student Performance and Understanding of Strategic Management Concepts: A Comparison of Techniques Transformational thinking in the Organizational Behavior Course: The Use of Metaphor as an Assessment Tool Creation of a Virtual Learning Community for the Global Virtual Enterprise Project Inter Institutional use of Educational Resources: Joint Use of Management Simulation Games Business Insights: Theory and Practice with the Aid of a Business Simulation Progress: An Experiential Exercise in Development Marketing The ABC's of teaching the Theory of Constraints to Undergraduate Business Students Demonstrating Principles of Organizational Purchasing Behavior through an Experiential Exercise The Virtual Manager: A Different Simulation for Managing Complexity When the Rules are changing and Chaos Breeds Innovation: Recapturing the Value of Constructive Thinking and Play in Simulation Training The Pitfalls, and Potential, of Actual Events as Problem Drivers Using Business Games to teach Environmental Awareness and Green Management: The International Experience with the ENSIM Game A Review of my ABSEL-Related Work Simulation of Government: A Workshop Using GEO Creating Internet-Based Games Using Perl and JavaScript Who's on First? Exploring the Concepts of Problem-Based Learning, Experiential Learning, and Lifelong Learning Students Learn Customer Service and Selling while Conducting Research So You Want to Run an NFL Football Team–An Honors Interdisciplinary Project Supervised Internship: The Employer's Perspective Using Computer Assisted Simulation to Teach International Business Strategy: A Case of the Multinational Management Game (MMG) Multi-Cultural Experiential Learning: A Computer Simulation in China An Appreciative Stance on Diversity as We Move into the 21st Century: A Timeline Exercise to Identify Key Experiences in Good Work Relationships between Black and White Peers A Day in the Life of an Interactive, Real Time, and Internet Delivered Course: A Demonstration Comparing Internet Search Engines: An Experiential Learning Exercise Total Enterprise Simulations and the Internet: Assessing Student Perceptions and Preferences Teaching Accounting Information Systems in a Practicum Format Providing Experiential Learning in Accounting through a Field Study Payroll Project A Spreadsheet Based Business Simulation Game Computer-Behavioral Simulations Training for Project Managers Simulation Scenarios - Rationale and Illustration