POTHOLES ALONG ROAD TO EVALUATING LEARNING OUTCOMES: THE CASE OF OUTDOOR MANAGEMENT TRAINING Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 202 POTHOLES ALONG ROAD TO EVALUATING LEARNING OUTCOMES: THE CASE OF OUTDOOR MANAGEMENT TRAINING Glenn M. McEvoy, Utah State University Paul F. Buller, Gonzaga University ABSTRACT Using experiences from an assessment of en outdoor management training program, this paper examines issues in training evaluation that are not often addressed. We present a five-step model suggesting key questions that must be answered before the appropriate evaluation approach can be determined. We conclude by warning that the clarion call for results-oriented training evaluation may he misguided, given the multiple and subtle purposes that training serves in the life of organizations. ISSUES IN TRAINING PROGRAM EVALUATION We identified five issues in training evaluation. To illustrate our discussion, we use the example of an outdoor management skills training (OMST) program in which we have been involved over the past two years. Work or “Perk’- The first issue is the extent to which the training program is truly an attempt to improve work performance or is in reality a perquisite for job performance already judged successful. Such objectives for the training program are very seldom spoken. To the extent that the true purpose of training is to reward good performers or to renew sagging spirits at company expense, an extensive performance-based training evaluation Is misguided. A simple reactions measure, or “smiles sheet,” may be all that is really necessary. OMST programs are necessarily off-site, and thus are potential candidates for “perks” rather than serious training. Participants in our OMST feel that it definitely is not a perk. A typical reaction is: “I’m glad I came, but I wouldn’t want to do it again.” For this reason, we decided that cur evaluation study had to go beyond the mere reaction level. Substance vs. Symbol. Considerable attention has been paid of late to the concept of corporate or organizational culture: shared values, rites, myths, symbols, and ceremonies. All organizational practices, including training, serve both to do (the practical side) and to say things (the symbolic side). Thus, training can be considered suggestive of the basic underlying culture or the “expressive side” of an organization. Off-site training programs like OMST are particularly likely to carry ritualistic or symbolic overtones. For example, selection to attend an OMST program can be seen as a symbol of the organization’s confidence in the individuals chosen. Program participants in our OMST case developed an informal network of program graduates that net occasionally, wore distinctive pins to work, and provided quick access to decision makers in other functional areas of the company. Completion of the OMST program essentially symbolized the opening of the gate to this extremely helpful informal network of prior program graduates. In fact, access to this network appeared to be one of the most beneficial outcomes of the program. External vs. Internal. A related issue is the extent to which desired training program outcomes are internal or external to the program itself. Authors have railed against the use of internal evaluation measures because they assume that all training must be designed to affect some external criterion. However, this is not always the case. For example, some training may be designed primarily to instill company pride, or perhaps inculcate company history and/or philosophy. OMST is again a potential example of this subtle expectation for training outcomes. Organizations sending employees to such training may be attempting to emphasize some unique aspects of corporate philosophy, such as the importance of individual risk-taking, or the balance between the need for individual excellence and group teamwork. Again, these changes may be unlikely to manifest themselves in behavior or results back on the job, so internal measures (e.g., diary entries of attitudes toward taking risks over the course of the program) may be appropriate. Behaviors vs. Outcomes. Once it has been determined that the purpose of training is truly work related and that the outcomes desired are more substantive than symbolic and more external than internal, the question becomes how to measure the changes back on the job. Basically, the issue here is whether to use a more “objective” quantitative criterion or a more “subjective” ratings criterion. It is not true, as is often assumed that results measures are superior to behavior measures. Improvement in participant skills are first order outcomes in OMST; results are second order outcomes. Many other potential causes exist for results besides managers’ skill levels. Further, in two years only 37 managers out of the total work force of over 6,000 have been trained in our OMST program. How great an impact on measurable organizational results can this small group be expected to have? Self vs. Other Ratings. Given the limitations of ratings from any single source, we suggested to our OMST client a Criterion involving combined ratings from multiple sources: self, supervisor, subordinates, and peers. We were acutely aware, in particular of the limitations of self-ratings when used by themselves. However, the client was more concerned about the time, cost, and organizational intrusiveness of collecting ratings from sources other than program participants. CONCLUSION Training evaluation research is not as straightforward as “textbook’ discussions would lead one to believe. Many subtle issues are not typically addressed; in fact, some of the issues are such that even the organizational sponsors of a training program may be unaware of their existence. Our concern here is that the clarion call for increased accountability in the training function may drive out all training that cannot demonstrate quantitative, measurable improvements in performance or results. Such an Outcome would be unfortunate indeed. Future training evaluation research needs to approach the problem from a broader perspective, possibly using ideograph qualitative research techniques, to capture important intangible changes that often accompany management training programs. Table of Contents Volume 17, 1990 The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - Revisited The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise Use and Effectiveness of an Analogy-Based Expert System Suggestions for Computerized Business Authors Dealing with Power: An Experiential Exercise Using Movie and Personal Diary Analysis Techniques A Model for Developing Student Skills and Assessing Outcomes Through Outdoor Training Computer-Aided Exercises Versus Workbook Exercises as Learning Facilitator in the Principles of Marketing Course An Exposition of Guilford's Si Model as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education Formal Planning and Simulation Team Performance: A Cross Sectional Approach Cases: Real Organizations in Real Time in the Classroom An Empirical Investigation of the Internal Validity of Marketing Simulation Game An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Game in a Principles of Marketing Course Factors Affecting Effective Teaching of Strategic Planning: Some Preliminary Evidence An Experiential Exercise for Learning About the Relationship Between Organizational Form & the Project Management Process Accounting Communication Skills can be Taught in the Auditing Course Modeling Cost Functions in Computerized Business Simulation: An Application of Duality Theory and Sheppard's Lemma A Life Cycle Analysis of Decision Making for a Strategic Management Team What's the Problem? A Dynamic Model for Teaching Problem Solving Skills Experientially International Currency Fluctuations: Money$im, A Simulation Superstores: A Specialized Retailing Simulation Within a Specialized Marketing Curriculum Factors Affecting Student Perceptions of Learning in a Business Policy Game VC + EL = VL The Name Game: An Experiential Exercise in Intergroup Relations The Effects of Experiential Accounting Work Experience on Student Performance in Intermediate Accounting Courses The Results of Using the Experiential Activity Group Performance Evaluation in a Business Policy Setting Using a Legal Database to Describe the Legal Environment of Marketing (and Business) Matching Environmental Uncertainty and Organizational Configuration An Instructional Computer Simulation of Tampering in QC Executive Evaluation of Student Learning in the Looking Glass Simulation Group Personality Composition and Total Enterprise Simulation Performance An Expert System for Selecting Analytical Techniques for Analyzing Marketing Research Data Effects of Cognitive Styles on Responses in an In-Basket Simulation A Psychometric Analysis of Kolb's Revised Learning Style-Inventory Selecting and Developing Experiential Exercises Using Movies Application of a Real-World Strategic Management Model in the Classroom Demand Equations which Include Product Attributes Consumption as the Objective in Computer-Scored Total Enterprise Simulations The Effects of Decision Format and Evaluation on Simulation Performance, Decision Time, and Team Cohesion The Effects of Computer Related Assignments on Student Performance in Business Administration The Money Game: A Dynamic Simulation Including Random Shocks for Money and Banking Courses Methods for Evaluating Performance on Business Simulations: A Survey The Effects of Synergogy on the Policy Course: Significant Improvements in Student Learning and Teacher Evaluation Conditions and Outcomes of Trust in a Two-Person Bargaining Exercise Bankgame Enhancing Computer Business Simulation with the Use of VGA Graphics An Experiential Approach to Entrepreneurship An Advanced Simulation Method (ASM) for Multiple Objective Problems The Influence of Experiential learning Techniques on Student Recognition of Non-Primary Learning Styles Negotiating Mergers and Acquisitions: A Cocktail Napkin Approach Identification of Unintended Effects in Experiential Laboratory Exercises An Experimental Comparison of Paper and Pencil and Computer Aided Decision Support Tools Porting a Simulation from the IBM World to the Macintosh World A Transaction Cost Analysis of Experiential Learning The Development of Experiential Exercises for Courses in Entrepreneurship and Small Business Management The Assessment Center as and Experiential Classroom Exercise Pricing Strategy Algorithms for Playing Business Simulations Organizational Structures for International Operations: An Experiential Activity Simulation Emphasis in the Business School Capstone Course An Integrated Approach to Computerizing the Business Curriculum Cross-Cultural Business Negotiations Exercise Organizational Socialization and Gender Differences in Students at Work Understanding Student Work Experience: A Content-Analytic Approach Introducing Executive MBA Programs with Management Games An Analysis of Improvement in Business Decision Outcome with Sequential Use of Two Simulation Games Teaching Business Policy Utilizing Mass Lecture and Individual Case Labs Potholes Along the Road to Evaluating Learning Outcomes: The Case of Outdoor Management Training Experiential Learning for Interior Design Students: Using CADD, Lotus 1-2-3, and Wordperfect Cognitive Learning Using a Computer-Based, Qualitative Interactive Business Simulation Sex Discrimination: Does the Woman get the Job or Does the Best Man Win? A Search for Visual Aids to Support Experiential Learning Through the 1990's An Experimental Analysis fo the Effectiveness of Student Role-Playing in Sales Training How to Have Students Learn from their Term Projects Self-Evaluation Exercise (SEE): An Assessment of Class Contribution A Study of the Influence of Team Formation on Attitudes and Performance in Management Games A Hardware Based CIM Simulation Laboratory Model Test Substance Abuse in Organizations Micro Computer Training Models Teaching Forecasting, Cash Budgeting and Inventory Model Building Using SBTools A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes An Adjunct Writing Instruction Assistant, The Computer; With an Illustration Classroom Software for ABC Analysis Utilizing Information Processing Technology to Enhance the Business Policy Simulation Experience Progressive Cases Realistic Job Previews Vs. Traditional Job Previews: Experiencing the Differences and Understanding the Consequences Investment Analysis Using the Pragmatic Multiplier Approach A Computer Simulation Interface for Competitive and Firm Analysis Self-Assessment of Ethical Decision Making Predispositions Preparing Managers for Overseas Assignments An Inquiry into Japanese Marketing: Workshop on Teaching Japanese Marketing The Performance Appraisal Feedback Interview: A Role Play for Human Resources Management Teaching Counselor Selling techniques Using Experiential Techniques to Teach International Topics International Management Simulation Gaming: Current Status and Future Developments A Realism Comparison of Simulation Technologies/Methodologies A Time-Efficient Game to Illustrate Concepts Taught in Management Courses and Management Development Programs Teaching the Management of Technology The Concept of Face and the Applicability of Experiential Exercises in an Oriental Culture's