A SYSTEMATIC APPROACH TO THE DEVELOPMENT AND EVALUATION OF EXPERIENTIAL EXERCISES Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 79 A SYSTEMATIC APPROACH TO THE DEVELOPMENT AND EVALUATION OF EXPERIENTIAL EXERCISES Gerrard Macintosh, University of Manitoba James W. Gentry, University of Nebraska-Lincoln Jeffrey J. Stoltman, Wayne State University ABSTRACT The paper describes a systematic approach to the development and evaluation of experiential pedagogies. The key notion is the need to determine the tacit knowledge required of successful people in the field. Once determined, the experiential exercise is developed to emphasize that knowledge, and the evaluation of the learning-taking place should be related to the student’s change in the ability to use that knowledge. A sales prospecting example is used to illustrate the process being suggested here. A SYSTEMATIC APPROACH TO THE DEVELOPMENT AND EVALUATION OF EXPERIENTIAL EXERCISES Building upon work in Instructional Psychology, Gentry, Stoltman, and Mehlhoff (1992) proposed the use of a more systematic approach to the design and testing of an experiential exercise. To some extent, the approach is not new, as most developers of experiential exercises go through a similar approach implicitly. What Gentry, Stoltman, and Mehlhoff (1992) suggested was that the approach be made more explicit. This paper discusses an attempt to develop an experiential exercise using the proposed approach. OVERVIEW OF THE PROPOSED PROCESS Glaser and Bassock (1989) listed three concerns critical’ to the development of instructional materials: first, attention should be given to isolating and describing competent performance in terms of both knowledge and skill. Second, both design and assessment should take the learners’ initial knowledge and ability into account. Third, we need to focus on the process of learning and explore how learning actually transpires. Gentry, Stoltman, and Mehlhoff (1992) also listed three somewhat similar stages involved in the design and testing of experiential exercises. We suggest that the two structures be combined to yield the four stages listed in Table 1. The purpose of this paper is to demonstrate how this process might transpire by discussing the design and use of a particular experiential exercise. Our organization of the rest of the paper will be structured around a stage-by- stage description of the process. The specific problem area dealt with is that of sales prospecting. PROBLEM AREA: SALES PROSPECTING Prospecting for new customers is important in any sales organization. This is true because it is the first step in the sales process, and the probability of success in the later stages depends to a large extent on how well the sales person has done in selecting potential clients on whom to call (Szymanski 1988). Recent research indicates that, on average, a sales call costs a firm about $200 (Business-to-Business 1988), which appears to be ample incentive for sales managers to minimize the number of calls made on unqualified leads or low potential accounts. Managers hoping to improve prospecting performance may benefit from gaining a better understanding of how sales people make prospecting decisions. Therefore, an interactive computer program was designed to investigate how sales people make such decisions. The forty subjects who participated in the study were sales representatives for a medium-sized life insurance company. This industry was selected for two reasons. First, sales people in this industry have primary responsibility for making prospecting decisions. In addition, it has been shown that there can be a considerable amount of variance among salespeople in this industry in terms of their ability to perform various selling tasks. STAGE 1: DETERMINING THE TACIT KNOWLEDGE NEEDED BY SUCCESSFUL PROFESSIONALS In this discussion, we focus on a subset of practical knowledge of particular importance, namely tacit knowledge. In the educational literature, there is a powerful impetus to discover the underlying aspects of intelligence, namely the lack of correspondence between performance in real-world pursuits and performance in school that is widely reported. In Neisser’s (1976) view, this situation exists because formal education tends to cover only a subset of what is required in the real world. Performance and pedagogy in education systems tend to reflect what he labeled academic intelligence, which differs from the intelligence required for performance in natural settings. While experiential pedagogy represents movement in the right direction on a theoretical level, absent a formal appraisal of the relevant tasks, which students are expected to perform with competence, problems in bridging the gap between real world and academic performances should be expected. Moreover, we assert that attempts to identify the basic processes and foci of experiential learning will continue to be thwarted until significant attention is given to the facets of practical intelligence. Though it is generally the case that the specific aspects of practical intelligence have not yet been identified, and the they will need to be developed on a case- or context-specific basis, conceptual guidance is provided by Wagner and Sternberg (Sternberg 1986; Wagner 1987; Wagner and Sternberg 1985, 1987). They define tacit knowledge as that which is not typically expressed or stated openly, not directly taught, or spoken about. It can be taught, but the ‘stuff is typically disorganized, informal, and relatively inaccessible, making it ill-suited for conventional methods of formal instruction. They key is not that it can't be taught or spoken, but that for the most part, it is not taught or spoken. There are three dimensions of tacit knowledge, and each dimension can be further delineated as shown in Figure 1. The content of tacit knowledge can pertain to managing oneself, others, or tasks. The contextual representation in tacit knowledge is either local (short-range, self-contained), or global (long-range, big picture). The orientation aspect of knowledge (representing both a behavioral and a motivational force) can be either idealistic or pragmatic. These facets of tacit knowledge have been tested and empirically separated from other identifiable aspects of knowledge (see Wagner 1987). Though largely confined to interpersonal skills and abilities, the distinction seems generally justified. Further efforts are needed to determine the extent to which these distinctions hold in other areas of competence. As to the use of experiential learning to provide this practical knowledge, Wagner and Sternberg (1987) comment that many experiential devices are not simulations of real-world tasks. Commenting specifically on the use of the “in-basket exercise to develop and test applied understanding, they offer the challenge which many ABSEL members (should) have experienced: This approach has considerable face validity (i.e., performance by the test is similar in appearance to the criterion performance), but it can be difficult to decide which aspects of a job to simulate and what standards to use in evaluating performance in the simulation (p. 438). Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 80 To apply the concept of tacit knowledge, we must begin by isolating the types of knowledge acquired and actually applied in real-world settings. As indicated above, we chose a narrow domain in which to investigate that knowledge. Rather than to consider the many facets of personal selling, we considered only the prospecting stage. Further, the task investigated was a somewhat novel presentation of sales leads, as it resembled the information display board format (most commonly seen in Consumer Reports’ comparisons of products across attributes). We acknowledge that this format is not the traditional manner in which insurance sales people receive sales leads, which is on a one-at-a-time basis. However, such formats are becoming increasingly common in the insurance industry as computerized data bases are used to centralize the early phases of the prospecting stage. In fact, one regional office of the firm under study did not participate in the study because they were already using such a computerized presentation of prospects, and thus their sales people had already acclimated to the type of stimulus used in the study (thus introducing an experience factor which had not been anticipated). The study involved 40 salespeople, split into higher performers and lower performers based on expertise in prospecting. The performance criterion was a linear combination of subjective ratings of prospecting ability by sales managers and of objective measures such as the number of new life insurance policies sold and the total dollar amount of new life insurance sold in the prior calendar year. The details of the study can be found in Macintosh (1992). Each subject was presented with a computerized format listing 12 prospects and eight attributes (income, age, occupation, etc.). Hall were told to rate each prospect and half were instructed to choose the best prospect. The MacIntosh laptop was programmed (using Hypercard) so that the sales person had to click the mouse when it was pointing at a specific cell in order to observe the attribute value for a particular prospect. The program recorded the sequence of information acquisition. In addition, the sales people were asked to provide verbal protocols (to think outloud) as they went through the process. The sales people later gave their ratings or choices using pencil and paper measures. Finally, the sales people in the judgment (rating) task were asked to choose their best prospect. The use of the computerized format allowed the measurement of both process and outcome. The results indicated that differences existed between higher and lower performing sales people. For example, for the sales people in the rating task, the evaluations were regressed against the attribute values, which yielded evidence that several attributes were weighted differently by higher performing sales people and lower performing sales people. More specifically, the study indicated that higher performing sales people put greater weight on the prospect’s income, occupation, and age, while lower performance put more emphasis on the number of goals and the number of children that the prospect had. This result suggested that lower performers had a narrower definition of the need for insurance, as compared to higher performers who had a broader definition that included needs related to higher income. An interesting interaction was found when the final 8 of 10 choices were investigated. When the sales people in the choice task were investigated, the majority selected either prospect 2 or prospect 7 (8 of 10 for high performers, 9 of 10 for low performers). However, when the choices were investigated for the sales people who first were asked to rate each prospect, a different pattern was found. For the higher performers, the majority (7 of 10) again selected prospect 2 or prospect 7; for the lower performers, only 4 of 10 selected either of those two prospects. We interpret these results as providing evidence that lower performers face more problems with information overload. The judgment task required that sales people investigate far more attributes per prospect; apparently the higher performers were able to sift through the information in order to select the best prospects, while the lower performers were more prone to get lost in the noise of the process. Thus, the study provided insight into the tacit knowledge needed by higher performers in terms of attributes to weight heavily and in terms of how to avoid problems of information overload. STAGE 2: DEVELOPMENT OF AN EXPERIENTIAL EXERCISE The same format used to measure sales people’s tacit knowledge of prospecting was used as an experiential exercise with sales management students. The cover story was in essence the truth: the computerized format was developed as a training exercise for an insurance company and was based on information commonly available to sales people. The primary benefits of using the same program are that it allows the labeling of "optimal" selections (prospects 2 or 7) and that the protocols provide a rich basis for insight into the thought processes of the successful performers undergoing the same task. STAGE 3: FOCUS ON THE PROCESS OF LEARNING Students without insurance sales experience will have a limited perspective as to what constitutes a *good prospect. Using the format to experiment and investigate possibilities is enjoyable for the students, but there is also some frustration once they realize that their intuition as to who is likely to buy insurance may not have any validity, material on prospecting for insurance clients was made available in the form of demographic profiles and discussions of both the needs and financial capacities of past buyers. The students are allowed to retry the exercise once their picture of a good prospect has been filled in somewhat. Feedback in the form of a summary of the process used to search for information about prospects was given to the students, along with comparable figures obtained from the higher performing sales people. Not only did they receive output in terms of the optimal clients, but also feedback about the cues that should be weighted most strongly. The feedback was provided after they had turned in a short summary paper giving their recommended choices and a discussion of the process that one should use to determine the chokes. The student’s explicit sell-assessment of what was learned was sought in the debriefing sessions following the use of the exercise. As Lederman (1992, p. 146) notes, Most simply stated, debriefing is a process in which people who have had an experience are led through a purposive discussion of that experience. The students’ directed reflections on the experience can raise a number of issues: --what attributes are most useful in evaluating prospects; --the need to use marketing research in order to identify better profiles of good prospects; --the tendency for some students to seek "similar" prospects (i.e., the younger prospects); —the fact that the most *similar prospects may not be the best* prospects; --the tradeoff between sales potential and perceived accessibility; --the frustration that certain information on prospects was not available; --that different types of processing take place when they are asked to rate all prospects as opposed to choosing one; --the screening process used to eliminate the majority of the prospects Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 81 from consideration; and --the possible dangers of using such an information base in prospecting. STAGE 4: THE MEASUREMENT OF WHAT 15 LEARNED The specific domain of concern here is sales prospecting, although the variety of issues arising in the debriefing session (as listed above) indicates that learning well occur in other domains as well. The use of the exercise allows detailed feedback about the process undertaken during the exercise, and not just outcome feedback. Further, they are given feedback as to the processes used by successful sales people in the same task. Contrasting the process used with that of high performers provides a vivid portrayal of the type of tacit knowledge held by successful sales people. Further example protocols from the data collection in Stage I can be provided during the debriefing session so that a clearer picture of the needed tacit knowledge can be obtained. SUMMARY This paper discusses a systematic approach to the design and testing of experiential exercises, using an information display format in a sales prospecting task as the context. The key addition to the usual approach is the overt attempt to measure the tacit knowledge held by successful professionals and then to structure the exercise so that students learn similar processes. The use of the four stages advocated here may well have some very positive implications for a closer integration of teaching and research. If one’s research investigates the processes used by successful practitioners, one can blend the findings into structured education experiences that provide students exposure to the tacit knowledge needed in the profession. REFERENCES Business-to-Business (1988), Sales and Marketing Management, 140 (November 8), 27. Gentry, James W., Jeffrey J. Stoltman, and Carol E. Mehlhoff (1992), "How Should We Measure Experiential Learning?: in Jerry 1. Gosenpud and Steven Gold (Eds.), Developments in Business Simulation and Experiential Learning Vol. 19, 54-57. Glaser, Robert and Miriam Bassok (1989), Learning Theory and the Study of Instruction, Annual Review of Psychology, 40, 631-666 Lederman, Linda Costigan (1992), Debriefing: Toward a Systematic Assessment of Theory and Practice, Simulation & Games, 23 (June), 145-160. Macintosh, Gerrard (1992), judgment and Choice in Client Selection: An Analysis of Decision making in the Prospecting Stage of the Selling Process, Unpublished Dissertation, University of Nebraska. Neisser, Ulrich (1976), General, Academic, and Artificial intelligence, in L. Resnick (ed.), The Nature of Intelligence, Hillsdale, NJ: Lawrence Erlbaum Associates 135-144 Sternberg, Robert J. (1985), Intelligence Applied San Diego, CA: Harcourt Brace Jovanovich Szymanski, David M. (1988), Determinants of Selling Effectiveness: The Importance of Declarative Knowledge to the Personal Selling Concept, journal of Marketing, 52 (January), 64-77. Wagner, Richard K. and Robert J. Sternberg (1987), "Tacit Knowledge in Everyday Intelligent Behavior, journal of Personality and Social Psychology, 52, (No. 6), 1236-1247. Wagner, Richard K. and Robert J. Sternberg (1985), “Practical Intelligence in Real-World Pursuits: The Role of Tacit Knowledge, Journal of Personality and Social Psychology, 49 (No. 2), 436-458. Wagner, Richard K. and Robert J. Sternberg (1987), Tacit Knowledge in Managerial Success, journal of Business and Psychology, 1 (No. 4), 301- 312. Table of Contents Volume 20, 1993 Dominant Personality Types and Total Enterprise Simulation Performance Shelf Wars: A Grocery Channel Simulation Shared Cultural Perspectives: An Experiential Exercise Utilizing International Students to Globalize the Classroom An Instrument for Investigating the Effectiveness of Teaching Methods in the Business Policy and Strategy Formulation Course Providing Better Trained Graduates for Accounting Employers The Ambition Gradient Approach to Evaluation of Computer Simulation Game Team Performance Alphatec: A Negotiation Exercise with Logrolling and Bridging Potential Using the Ideafisher Idea Generation System as a Decision Support System in Marketing Strategy Courses A Dynamic Market Share Allocation Model For Computerized Business Simulations Multi-Cultural Adaptability Using Experiential Learning in a Graduate Course Development of Experiential Applications in HRM: Practicing What Preach and Preaching for Practice Linking Students and Business Leaders Through Portfolios Debriefing International Experiential Learning Exercises: Road Signs for Effectiveness Sales Manager: A Simulation Modeling Interactive Effects in Mathematical Functions for Business Simulations: A Critique of Goosen's Interpolation Reducing the Complexity of Interactive Variable Modeling in Business Simulations Through Interpolation Antecedent Biases of Experiential Learners: Trainee Occupation and Subgroup Diversity Pax in Terra Sancta: Simulating the Middle East Peace Negotiations A Multiple Regression Case In Experiential Learning Changes in Ethnocentric/Geocentric Orientation by Business Students after Exposure to a One Summer Course in International Marketing's A Systematic Approach to the Development and Evaluation of Experiential Exercises Entrepreneurs Evaluate Experiential Education A Linear Programming Approach to Open System Total Enterprise Simulations Reflecting Leader Behavior from the Looking Glass, Inc. Simulation Linking Cognitive Styles, Teaching Methods, Educational Objectives and Assessment: A Decision Tree Approach Restructuring Management Education in Post-Communist Countries: How Western Experts Can Help Managerial and Cultural Pre-Conditions for Superior Performance in a Global Setting: An Experimental Study with the Aid of Business Games Multiple Industries in Computerized Business Gaming Simulations Content or Process? - Content and Process! Some Observations and Reflections About Management Education in Central Europe Out-of-Class Experiences to Promote Volunteerism Enacting the Linguistic Consciousness of the Modern Managerial Mind: Post-Modernism and Experiential Learning Intergrating Experiential Exercises into the College Curriculum: The Case of Internationalizing the Business Curriculum Simulation Marketing Oversights Incorporating Advertising Creative Strategy into Computer-Based Business Simulations The Dynamics of a Partnership Between Business and Education Collaborative Education Done Globally Experiential Systems Analysis CADPLAN: A Simulation for Comparative Advertising A Doctoral Symposium: Preparing Students for Conference Behavior Comparing the Simulation with the Case Approach: Again! Total Quality Management: A Model for Continuous Quality Improvement The Quality Audit: An Experiential Exercise for Business Students Extending the Reach of Simulations: DECIDE Heads for the Inner City Lessons Learned from a Customized Management Development Simulation The Foreign Exchange Spot Trading Simulation Using Lotus 1-2-3 to complete a Triple Play in a Simulated Competition International Business Education: Is Enough Being Done? Matching of Student-Teacher Cognitive Style as a Factor in Student Success in an Introduction to Information Systems Course Breathing (More) Life into the Case Approach Lord of the Flies: A Live Case Approach to Leadership Cooperative Case Studies: Experiential Tools for Teaching Business Problem Solving Tools Strategy Simulations in Context: An Evaluation of Key Dimensions The Distribution Channel Game Evaluation of a Simulation Game as an Education Tool for Utility Professionals The Relationship Between Total Enterprise Simulation Performance and Learning Total Quality Management does not Happen by Magic, but it can be Taught Using a Pedagogical Methodology that Utilizes Magic Effectively Preparing Students for Careers in a Global Environment by Integrating Total Quality Management Thoughout the Business Curriculum An Empirical Investigation of Cognitive and Performance Consistency in a Marketing Simulation Game Environment Using MARSGAP with LAPTOP: (A Marketing Simulation Game Analysis Program) with LAPTOP: A Marketing Simulation Adapting TQM Implementation to Organizational Level An MBA Business Simulation: Executive Interaction Experiential Exercises and Pedagogy Track Workshop: Experiencing Cultural Diversity in the Classroom (and the Hotel Meeting Room) Closing the Gap between Corporate and National Culture The Dynamic Manufacturing Company The Use of Experiential Techniques in Corporate Training The State of Simulation Gaming in Easter European Countries- Principally Russia An Experiential Exercise in Cross-Cultural Training Valuing Differences: A Conceptual Framework Demonstration of an Experiential Exercise Effectively Using Experiential Learning to Impart TQM Concepts in a High Technology Environment The Older Worker Questionnaire: An Exercise Concerning Older Worker Stereotypes and Behaviors The Crime Fighting Task Force: An Exercise in Organizational Politics Welcome to the Party! An Expression of Vocational Preference Experiential Exercise for Imparting Cross-Cultural Appreciation Six Swift Simulations on Globalization Overview of BASF Delegate Program