MEASURING THE EFFECT OF AN EXPERIENTIAL EXERCISE New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 277 MEASURING THE EFFECT OF AN EXPERIENTIAL EXERCISE Donald J. Messmer, College of William and Mary Murphy A. Sewall, State University of New York at Albany INTRODUCTION The purpose of this study is to describe and demonstrate a technique for measuring the impact of an experiential exercise on the participants’ perceptions of and preferences for various career alternatives. The method utilizes an experimental design with measurement of perception and preference obtained by means of a multidimensional scaling technique. As a demonstration of the method, it has been used to assess the effect of a personal selling role-playing exercise. Motivation for this study was stimulated by the evidence that in recent years students have become increasingly disdainful of accepting jobs as salesmen. In part, this trend seems to be the result of a failure by marketing academicians to effectively communicate the diversity of skills required to be effective in a personal selling role. Student attitudes toward selling are often formed by their own experiences with sales people, or on the basis of the information derived from a marketing course. Unfortunately, the sales model with which most come in contact is the retail sales clerk, a role which rarely demonstrates the range of skills needed in a more complex selling environment. Nor is the image conveyed by these models dispelled by the usual marketing curriculum which is trending toward the development of managerial abilities and the teaching of theory [3, 5]. As a result, the student is left with the perception that selling is a low status task requiring few skills. Since the purpose of an experiential exercise is to increase student understanding of basic principles and the use of these principles, in a specific situational context, it seems feasible that inclusion of a personal selling role exercise in an initial marketing course might alter participants’ perception of the selling task. Further, since experiential exercises are useful in developing skills in the types of situations which frequently occur in selling, the role exercise should add to the fuller development of a student in marketing [4, 7]. It is less clear, however, whether changes in perception of the selling task through exposure to an experiential exercise will also alter preferences for sales as a career. Hence, what is needed is a means of directly observing career perceptions and preferences in a controlled environment. THE ROLE EXERCISE The exercise which was designed for the experiment puts the student in the role of selling an industrial New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 278 adhesive, called Brunswick 910. The salesman was asked to role play a sales call on the buyer for a sporting goods manufacturer who produces a line of laminated wood tennis rackets. Preliminary contacts made prior to the role situation have established that the buyer has an application for the adhesive, and that Brunswick 910 does have certain advantages. These advantages, however, were not all equally important to Marlite Co., the sporting goods producer. Further, the salesman was given a range of prices and the buyer had an existing source at an equitable price, although the student salesman was not told the price. Students were advised that a portion of their grade in the course would depend on how well they perform in the exercise. Performance was evaluated by calculating a net profit for their order (or saving in the case of the buyer), the amount of which depended critically on the price agreed to by buyer and seller, and on the size of the order (if any). Of course, profits to the salesman were generally achieved only at the expense of savings to the buyer. METHODOLOGY Two sections of a Principles of Marketing course at the College of William and Nary were asked to participate in the role exercise described above. They were told it was an experiential exercise to familiarize them with the personal selling task. The research design was an after-only study with two experimental groups and a control group. The two experimental groups were obtained from the first section by randomly pairing all members of the class into buyer-seller dyads. The second section of the course formed the control group. In order to avoid an effect due to cultural role expectations, both members of a dyad were of the same sex. At the conclusion of the exercise, which normally required about 45 minutes, the subjects in the experimental groups were asked to fill out a short questionnaire detailing the agreement they had reached, and describing their attitude toward their counterpart in the dyad, and toward the product they were selling (or buying). In addition, all students (both experimental and control groups) were asked to complete a “Survey of Student Perceptions of Selected Employment Opportunities.” This questionnaire consisted of nonmetric similarity judgment requests in the form of anchor point rankings among ten job titles which are commonly held by BBAs immediately after graduation. The jobs were: bank officer, sales representative, programmer/systems analyst, accountant, office manager, market researcher, purchasing agent, financial analyst, insurance underwriter, and management trainee. The students were also asked to rank order the ten job titles according to their overall preferences for them. Completed questionnaires were obtained from nineteen students who played the role of buyer, seventeen who played the role of seller and eighteen (control) students from the second section. These data, which formed the basis for scaling perceptions, consisted of establishing New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 279 each of the ten job titles in turn as a reference and requiring respondents to rank the remaining nine titles according to their similarity to the reference job (ten rows of nine rankings). Similarity data obtained in this manner has the joint benefit of minimizing demands on the respondent, of allowing the subject to determine the appropriate frame of reference for general similarity and of effectively disguising the purpose of the questioning. ANALYSIS OF STUDENT PERCEPTIONS Each of the three groups (buyers, sellers, and control) was scaled in three dimensions according to Young’s asymmetric multiprocess, individual difference model using the ALSCAL multidimensional scaling program [6]. The adequacy of the group spaces produced by this algorithm was tested by computing a row conditional rank order matrix based on the three-dimensional distances in each group space and comparing these matrices with the student input matrices using Kendall’s rank order correlation coefficient. Forty-eight of the original fifty-four input matrices were significantly correlated with the appropriate group space (at the five percent level--eighteen of the nineteen buyers, fourteen of the seventeen sellers, and sixteen of the eighteen controls). Certain consistencies existed between the multidimensional perceptual relationships across all three group spaces. The financial analyst and accounting job titles were consistently perceived as highly similar. To a lesser degree the titles of sales representative, purchasing agent, and management trainee were perceived as related. The multiprocess model also estimates weights for each stimulus on each dimension (as well as differential weights for each subject on each dimension). The stimulus weights indicated that the sales representative position loaded on the second dimension for the control group but on the third dimension for the two role-playing groups. While statistical tests of the significance of this effect are not available, the qualitative indication is that the position of sales representative is associated with a less salient (important) dimension for those students who participated in the role-playing exercise than for those students who did not participate. It is possible that the exercise was actually counterproductive in enhancing students’ perception of a sales position as a viable career opportunity. In order to make direct comparisons between the responses of all three groups of respondents, it was desirable to reorient the three group spaces into a common coordinate structure. The possibility of accomplishing this objective seemed plausible due to the evident visual similarity of the three group spaces. The CANDECOMP (N-Way Canonical Decomposition) option of Carroll and Chang’s New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 280 INDSCAL (Individual Difference Scaling) program provides a procedure for combining a number of similar coordinate matrices into a single configuration [2]. Canonical decomposition allows for differential rescaling of axes (stretching or shrinking) as well as orthogonal rotations. After conversion to a common configuration, the INDSCAL program indicated an average correlation of the original three spaces to the common space of ninety-six percent (ninety-four or buyers, ninety- seven for sellers, and ninety-seven for controls). However, a row conditional rank order matrix derived from the three-dimensional distances of the combined group space compared to the fifty-four original student matrices produced only twenty-four statistically significant Kendall’s rank order correlation coefficients (fifteen of nineteen buyers, nine of seventeen sellers, and none of the eighteen controls). The failure of any of the control subject matrices to correlate with the combined space tends to support the earlier indication of a difference between the perceptions of the control group and the role-playing groups. The canonical decomposition analysis does not provide an obvious explanation for this result, and no statistical procedures for investigating specific differences between the groups are currently available. Even though there may be some imperfections in perceptual interpretations, the combined group space is a useful measurement system for analyzing job preferences across groups. Chang and Carroll’s PREFMAP (Preference Napping) program was used to determine “ideal points” for each of the fifty- four students in the combined group space [1]. The distances from a subject’s ideal point and each of the stimuli (job titles) express the relative preferences of the subject for the stimuli, with smaller distances implying greater preference. Distances between each ideal point and the stimuli were compared to the original preference rankings for each subject using Spearman’s rank order correlation coefficient. Thirty-eight of fifty-four coefficients were statistically significant (at the five percent level-- fifteen buyers, ten sellers, and thirteen controls). Figure 1 contains a plot of the first two dimensions of the combined space containing the ten job title points and ideal points for the average rankings of each of the three groups (all three of these average ideal points are significantly correlated with the average group rankings). The average ideal points all appear in closer proximity to the more quantitative job titles (which are plausibly proximate to each other)--bank officer, financial analyst, accountant, programmer/systems analyst, and market researcher. Sales representative and purchasing agent appear close to each other, as one might expect, in the upper left quadrant. The fact that insurance underwriter appears by itself and closer to sales representative or purchasing agent than to the quantitative positions is an anomaly which may indicate that these students do not fully understand the difference between an underwriter and an insurance salesman. New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 281 TESTS OF THE RESEARCH HYPOTHESES The mapping of the students’ rank order preferences for the job titles into their individual metric spaces provided data appropriate for an analysis of variance of the relative preferences for the jobs across the three groups. Further, analysis of contrasts may provide a direct test of the hypothesis that members of the two role-playing groups are more favorably disposed toward the sales representative position than members of the control group. As a preliminary to this analysis, contingency table analyses of the distributions of other factors across the three groups were conducted. Since the experimental design did not randomly assign subjects to role-playing and control groups, it seemed prudent to search for evidence of plausible sources of bias that might result from the actual make-up of the groups. New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 282 Tests of the significance for the different locations of the subjects’ ideal points in the combined group space logically ought to exclude those students whose ideal points are not significantly related to their original preference orderings. It was earlier established that the combined group space is an imperfect representation of the perceptions of members of the control group in particular. Therefore, it seemed prudent to examine whether or not the significance of subjects’ ideal points for preference in the space is related to group membership. The chi-square value for this test was only 1.787 (tail probability of .41 with 2 degrees of freedom). Hence, an obvious problem with proceeding did not appear. Several other tests were also undertaken to investigate possible systematic differences between the members of the three groups. No significant differences were found on the basis of age (64 percent were twenty), sex (61 percent were male), major (43 percent accounting , 48 percent management, 6 percent economics), prior selling experience (only 4 percent had none, 50 percent had limited experience, 35 percent reported moderate experience, and 11 percent had extensive experience), or reason for enrolling in the course (80 percent were required to take the course). These findings were not altered when only those subjects whose ideal points in the combined space were significantly related to their preference orderings were considered. Unfortunately, analysis of variance also failed to detect any significant differences in the average ideal point distances to any of the ten job titles across the three groups. Analysis of contrasts between the two role-playing groups and the control group as well as between the three possible pairs of groups also failed to yield significant differences for any of the ten jobs. Since twenty of the thirty-eight respondents with significant ideal points in the combined group space had selected technical majors (economics or accounting), the analyses of variance and contrasts were rerun with the remaining eighteen subjects. There were no noticeable changes in the results. While the outcome of the hypotheses tests was frustrating, an alternative, less sophisticated approach also failed to detect a significant relationship between attitudes toward personal selling and group membership. Contingency table tests were undertaken on the level of agreement or disagreement with the following concepts: - Personal selling is interesting, - I would be proud to be a salesman, - Salesmen are well paid, - Salesmen are honest, - Good salesmen are born, and - Salesmen make a positive contribution to society. New Horizons in Simulation Games and Experiential Learning, Volume 4, 1977 283 While this study failed to detect a significant difference between the role-playing subjects and a control group of similar students, there are indications that this finding is the result of an inadequacy in the role- playing exercise rather than a failure of the methodology. This tentative conclusion is based on the apparent reduced salience for the sales representative position for the role players compared to the control subjects in the perceptual analysis and the fact that the average ideal point for the control group is closer to the sales representative position (not significant) than for either role playing group. It is possible that the relatively high fraction of participants in this research with at least some prior selling experience constituted a population of subjects with attitudes unlikely to be altered by the brief role- playing experience. Further, the role exercise might have overemphasized the price negotiation to the exclusion of other skills. The multidimensional scaling methodology employed in this study appears promising as a means for evaluating student responses to pedagogical instruments. In this case, the analysis questions the further utility of this particular role playing exercise. Additional support for this conclusion would follow from a demonstration of the methodology’s ability to detect significant differences between a control group and participants in a more effective exercise. REFERENCE S 1. Carroll, J. D. and J. J. Chang. “Analysis of Individual Differences in Multidimensional Scaling via an N-Way Generalization of ‘Ecart-Young’ Decomposition,” Psychometrika, Vol. 35 (September, 1970), pp. 283-319. 2. Chang, J. J. and J. Douglas Carroll. ‘How to Use PREFMAP and PREFMAP2 - Programs Which Relate Preference Data to Multidimensional Scaling Solution,” unpublished paper, Bell Laboratories, Murray Hill, New Jersey, 1969. 3. Harding, David K. Address before American Marketing Association, Houston, Texas (August, 1972). 4. Kolb, David A., Irwin M. Rubin and James M. McIntyre. Organizational Psychology: An Experiential Approach. (Englewood Cliffs, New Jersey: Prentice-Hall, 1971). 5. Peterson, Robin T. “A Return to Personal Selling in the Marketing Curriculum,” Proceedings, 1976 Educators Conference, (Chicago: American Marketing Association, 1976). 6. Talcane, Y., F. W. Young and J. De Leeuw. “How to Use ALSCAL’ a Multidimensional Scaling Program with Table of Contents Volume 4, 1977 Double Play for Gaming Effectiveness Adaptive Rule Changes in Computer Simulation Gaming Œ A Means of Pedagogical Reactive Interchange Monte Carlo Simulation in Personnel Management Training Teaching About the Implementation of Job Redesign Using Simulation and Group Discussions An Interactive Simulation of Private Sector Collective Bargaining Leadership Evaluation and training through Behavioral Simulations: Method, Results and Future An assessment of the Effect of Experiential, Simulation and Discussion Pedagologies Used in Laboratory Sections of an Introductory Management Course An Experiential Understanding of the Trust Dimension Using Consulting Cases to teach Business Policy An Experimental Testing of Teaching Methodologies in Marketing Interpersonal Skill Development: The Experiential Training Unit (ETU) and Transfer of Training An Analysis of the Relationship between Personality characteristics and Preferred Styles of Learning Analysis of Effective Communication Skill Development in Graduate Business and Engineering Experiential Education Changing Perceptions of Learning in a Simulated Environment Student Perceptions: Simulation and the Corporate Policy Course Degree of Uniformity in Achievement Motivation Levels of Team Member: Its Effect on Team Performance in a Simulation Game Channel Conflict, Cooperation and Control: an Experiential Learning Exercise Differences in Experiential and Non-Experiential Learners' Reactions to Conflict between Individual and Organizations Behavior The Evolution and Evaluation of a Required, Senior-Level Course in Experiential Business Applications Building Management Skills through Problem Solving A Live-Case Approach to the Business and Society Course Experiential Learning: Toward the Development of a Theoretical Base and the Identification of Variables and the Hypotheses to Guide Research The Role of the Administrator in Experiential Learning and Simulations Some Thoughts on a Theory of the Use of Games and Experiential Exercises Three Applications of the Management of Learning Grid An Analysis of ABSEL: Its Past Achievements and Future Prospects New Horizons in Simulation Research Prediction of Academic Achievement in a Simulation Mode via Personality Constructs Sex Differences in a Bargaining Simulation COM-GAME: A Commodity Trading Game for Use in an Introductory Business Statistics Course A Financial Institution Management Game with Direct Participant Interactions A Non-Computerized Marketing Planning and Strategy Game Delphi in the Classroom: A Demonstration of Forecasting Economic Activity The Potential of Programmable Calculators for Processing Small Business Simulations Can a Small Predominantly Clack University Incorporate the Computer Simulation Gaming Teaching Methodology into it Curriculum Measuring the Effect of an Experiential Exercise Experiential Learning - Analysis of a Partial Success Predicting Participants' Performance and Reactions in an Experiential Learning Setting: An Empirical Investigation A Simplified, Non-Computerized Marketing Channels Game Manufacturers and Retailers: A Negotiation Game for Beginning Management Students Petroleum Management Game A Securities Dealer Simulator SIM ECO SOC with Business Curriculum Modules: A Simulation for Business Ethics and Morals The Picnic: A Perceptual Errors Exercise Salt III; an Experiential Exercise to Highlight the Interpersonal Dynamics Involved in the Negotiation Process Experiential Exercise on Values, Attitudes and Conflict Resolution in Organizational Behavior Kick'N Go: A Product Management and Social Responsibility Dilemma The Use of Self-Assessment Work-shops in a school of Business Administration The Dilemma of Self-Perception