EXECUTIVE EVALUATION OF STUDENT LEARNING IN THE LOOKING GLASS SIMULATION Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 128 Executive Evaluation of Student Learning In The Looking Glass Simulation Diana Page, The University of West Florida Ralph M. Roberts, The University of West Florida ABSTRACT This study measures the effects of involving beginning management students in a management simulation, The Looking Glass. A control and experimental group of students were compared in a pre and posttest situation. They responded to the question, what does a manager in a mid to large organization do on a daily basis? Executives of mid to large organizations and professors evaluated the student data. Data analysis indicated significant gains in student knowledge between the two groups. INTRODUCTION The activities in which a manager is actually engaged on a daily basis are varied, and often fraught with incomplete information, ambiguity, conflicts, time limitations, and a sense of urgency (Mintzberg, 1975). A never ending number of Items and activities demand attention and must be prioritized, delegated and attended to through planning and decision making. Management teaching must bring a realistic view of the managerial situation to the classroom so that students can obtain an idea of what a manager actually does on a daily basis and can begin to learn how to deal with complex managerial roles. Teaching requires more than just presenting the functional aspects and theories of management. Practicing managers state that institutions of higher learning produce individuals who are intelligent in “book learning” -- that they can discuss concepts and theories but do not understand the complex balancing of the process of management and do not understand what managers really do. Richard K. Wagner and Robert J. Sternberg (1987), further Indicate that It Is not enough to know what a manager should do, but also that one must be competent to perform those functions. An important step toward competency, beyond the knowledge of vocabulary and theories, is for students to obtain an accurate view of managerial activities. This paper reports a rigorous study of the educational benefits of a complex managerial simulation. The experiential learning event, The Looking Class is evaluated to determine if a student can learn what a manager does on a daily basis in a mid to large sized organization. Experiential learning is a recognized methodology which can affect the learner in three ways: 1) the learner’s cognitive structures are altered, 2) the learner’s attitudes are modified, and 3) the learner’s repertoire of behavioral skills is expanded (Johnson and Johnson, 1982). These outcomes are most likely to occur if learners are aware of content matter, attitudes, social environment, behavioral patterns and can develop introspective capabilities to gain an adequate knowledge of themselves. To learn what a manager does, for example, the learner must develop a concept of the functions of management, a supporting vocabulary, an action theory concerning the process of management, an understanding of the nature of managerial work, a perception of actual activities of managers on a daily basis, and an assessment of self to determine whether s/he must develop additional skills. Business simulations, such as the one used as the treatment for the experimental group in this paper, The Looking Glass, (Lowbardo, McCall, DeVries, 1983) are designed to provide a realistic simulation of managerial activities which will affect the learner in the three ways mentioned above and, in addition, will assist students in becoming aware of what managers do on a daily basis. The business simulation, The Looking Class is carefully designed to offer participants a “gut feel of managerial life, knowledge of managerial work-- its pace, pressure, ambiguity, variety, and complexity’ (Lombardo, et al, 1983). LITERATURE REVIEW Much has been written during the past decade about the relative worth of simulation games as educational tools. Brenenstuhl (1975) reports two reasons why business professors became interested in simulations: first, experiential learning offers a dynamic environment through which students can study behavioral aspects as they might occur in organizations and second, that active involvement enhances the learning process. Anderson, P.H., Lawton and Leigh (1988) describe the problem of evaluating student performance, and utilize Bloom’s taxonomy to facilitate the evaluation process. This current study provides more realistic data by using executives as judges to evaluate student data. A. Parasuraman (1980) acknowledges the enchantment of simulations to he the ‘hands on” or participatory nature of the exercises. One concern of Parasuraman, however, is the need to identify the knowledge supposedly imparted through participation in the simulation. Gosenpud (1988) supports this assertion and suggests that the experiential method of teaching may be shown to he sore effective if care is taken to design a test, which matches the goals of the learning experience. For this study the extent to which the simulation accomplishes the objective of providing greater realism about managerial work to students is measured by asking managerial practitioners to evaluate student learning from the exercise. Gentry, et al (1984) discusses the advantages and disadvantages of testing simulation gases to investigate their merits as teaching devices. The advantages include controls to ensure internal and external validity, minimal costs, and the removal of sensitivity associated with real world activities. These advantages are apparent in this study. In general, the literature suggests that experiential learning and simulation games can offer Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 129 the Students an opportunity to observe, react and reflect 8B they might in real world situations. On the other hand, rigorous study to identify what is actually taught by simulations is difficult. The objective for this study, then, is to determine if there is true learning about the nature of managerial work expressed in student papers and as recognized by practicing executives from the real world environment. TUE SIMULATION EXERCISE The Looking Glass is a complex in-basket exercise, which creates a day in the lives of the top twenty managers of a mid-sized manufacturing corporation. Student participants are free to call meetings, write memos and take (or defer) decisions. Like any company, Looking Glass has its share of problems ranging frog the trivial to the titanic. These total over a hundred, and participants may deal with them as they see fit. Problems and issues cover many areas, including finance, personnel, legal matters, production, sales, research and development, and safety. Examples include the following: *An opportunity to acquire a new plant, *A decision about what to do with a plant that has lost money for the last few years, *Supply shortages, *Labor unrest, *Production capacity limits, *Competition with foreign manufacturers, and *The need to fill a vacant plant manager position. There are three divisions in Looking Class, each of which has a different external environment. The Advanced Products Division exists in an unstable, highly volatile environment; the Commercial Class Division operates in a reasonably stable, predictable environment characterized by high-volume, low-margin products and well-established customer relations; and the Industrial Class Division exists in an environment containing both unstable and stable products Lombardo, et al (1983). METHODOLOGY The purpose of the research is to determine if the exercise, The Looking Glass provides learning to change students’ perceptions toward a more realistic view of the day to day activities of a mid level manager in a large organization. A mid level manager was considered to he one who holds a position tvo to three positions above the supervisory level in an organization. A large organization, for purposes of this study, vas defined as one which has more than 500 employees at the site. The Looking Class has been used in .any educational settings; this research focuses on college students in a typical classroom setting, “playing” the situation over two lengthy classrooms periods of three hours each, The instructors selected students for each of the simulation roles with an eye to maximizing the educational benefit for all. The experimental group undertook the simulation, while the control group continued its usual classroom activities which were lecture, discussion and some small group role playing. Pre and post tests were given to both groups by asking then to write one page response to “What does a manager in a mid to large organization do on a daily basis?” Practicing mangers in mid to large organization were asked to evaluate the accuracy of the randomly mixed student papers. The class instructors also evaluated the student responses. A one way ANOVA with repeated measures was used to evaluate the results. The Sample Two management fundamentals classes were selected At this university. The total number of students in the sample is 58. There are 26 in the control group (Group A) and 32 students in the experimental group (Group B). The groups were demographically similar with ages from 22 to 36. They are juniors and seniors whose majors are predominantly business disciplines but also include arts and sciences, and education. Approximately 1/4 of these students are management .majors. Hypotheses The following hypotheses (H) were established: H1 Students in group A will change perceptions of managerial work to conform with executive judgement to a greater extent than will students in group B. H2 There will he agreement among executive ratings (this hypothesis was established to investigate the interrater reliability). Data Collection Students in the two management classes were given an assignment to write 8 short description of their view of “a day in the life of a mid level manger in a mid to large organization.” This was collected by the class instructors. Subsequently, one class, group A, (the experimental group) was provided “Looking Class” for approximately six hours as an educational intervention to acquaint thee with the managerial day This exercise has been carefully developed and validated as an appropriate view of management (McCall and Lombardo, 1979). The second class, group B, (the control group) did not hive the exercise, but instead continued their usual classroom activities for two weeks. Both classes were then asked to write for a second time their view of “a day in the life of a manager in a medium to large organization.” Judges to evaluate the students; papers were obtained from two sources, 1) outside executives who held managerial positions in medium to large organizations as defined in the study. The positions these judges held were a) the top level engineering (management) position in a federal agency, h) an upper level position in the main office of a large electric company, c) an upper level position in a large plant of a multinational industrial company, and d) an upper level position in a multinational energy company. The second source of judges was the university setting and here the two researchers, who were teaching the class, were also the evaluators of the student papers. An evaluation form was designed as a guide in assessing the accuracy of the students’ descriptions of managerial activities. The fore provided the following definitions for scoring: Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 130 1. This description is almost or totally inaccurate. 2. This description is generally inaccurate with some inaccuracies. 3. This description is somewhat sore inaccurate than accurate. 6. This description is in between; about equally accurate and inaccurate. 5. This description is somewhat more accurate than inaccurate. 6. This description is generally accurate with some inaccuracies. 7. This description is almost or totally accurate. The six judges scored the student descriptions of “a day in the life of a mid level manager in a large organization,” blind; that is. all papers were coded by a student assistant, copied, and randomly mixed so no evaluator knew the origin of any description. The executives were asked to score the descriptions in accordance with the executive’s view of the daily managerial life in their organization, as did the professors - Data Analysis The experimental design consisted of a one way ANOVA with repeated measures, that is, the before and after treatment conditions of the experimental and control group were evaluated by each of the six judges. A preliminary review of the data indicated questionable results from one judge. For the control group, the mean differences by judge were the following with a plus value indicating g change from a less to a more accurate view of managerial activities. Scores for the control group were -.44, -.82, -.89, -.44, -2.78, .33 and for the experimental group 1.96, .76, .81, .83. -. 69, 1.11. Ratings from judge 15 were noted to he greatly different from the other ratings. Further investigation indicated a likely misunderstanding of the rating scale so the set of scores from this judge was deleted from the analysis. With the five judges, the average change In the control group was -.45 and in the experimental group +1.10 and a significant difference in the mean change in student perceptions was found (F(1,37) 18.36, p< .01). Thus there is a significant positive change for the students in the experimental group in contrast to the control group. The hypothesis that there was agreement among judges in scoring the same must he rejected (F(4,148) 3.40, P( .01). This lack of reliability would indicate a lack of uniformity in scoring student papers, which would cast doubt upon the test of the first hypothesis. A further review of the mean differences by judges for the two groups revealed the following: Again, the mean differences for the control group were -.44, -.82, -.89. -.44 and .33 respectively and for the experimental group 1.96, .76, .81, .82, and 1.11 respectively. Judges one and five were the professors teaching the crass who also evaluated the student papers. One of the professor’s mean difference was out of line with the executive evaluators in the control group (.33). In the experimental group both professors mean scores indicated the greatest difference in the positive direction of greater student learning. In addition one professor’s mean difference score (1.96) was more than double all the outside executives’ scores. It appeared the professors scores were considerably different from the executives. Given this variation, the data were analyzed using the executives’ scores only. The average .can difference for the three judges for the control group was -.70, indicating a possible decline in the accuracy of the control group’s view of the managerial world. For the experimental group the average mean difference for the three judges was +.74 indicating an increasingly accurate view of managerial activities. With the three judges the hypothesis of agreement among the judges in scoring could not be rejected (F(2,78) = .72, P<49). The executive judges were remarkably uniform in differences of before and after scores for the student papers. In summary the five judges provide a significant difference in scores in the direction of more realism about the work of a manager in the experimental group. However, the five judges had significant differences in their scoring of the change in student papers. The two professors appeared to he causing this difference among evaluators and when they were removed no significant differences in the remaining executive evaluations were found. With the three remaining, highly consistent, executive evaluators a significant difference was again found in the changes of student scores in the control and experimental group. DISCUSSION There is a continuing need to bring information to the classroom which will provide aspiring managers realistic learning about managers’ activities on a day to day basis. It is no longer sufficient for students to learn only the management vocabulary and theories; they must understand and translate those ideas and concepts into “bottom line” learning. This approach provides a “realistic job preview” for these aspiring managers, which will translate to greater job satisfaction for the practitioners and greater retention for the hiring organizations. The benefits of providing students with The Looking Glass simulation is threefold. Participants are better able to determine their fit in a managerial environment. Further they are better equipped to build upon their insights with additional business related coursework. Finally, they will be better able to translate their academic knowledge to the managerial environment. An interesting conjecture, from the closeness of the executives’ scores, is that there actually may be some uniformity in the world of executive managerial activities. The three executive evaluators were remarkably close in their evaluation of the student papers. Perhaps a day for these managers is similar interest of the process and acts undertaken. The hurry-up, disjointed, discontinuous, ambiguous environment found by Mintzberg may be accurate for these executives thus allowing them to evaluate the change (difference) in student papers rather uniformly according to the 7 item scale provided. The professors’ evaluations showed a marked improvement for the experimental group. It appears that the instructors, knowing The Looking Glass exercise were unconsciously keying in on the words, phrases and descriptions which might have cone from the exercise. This would account for the much greater change in score for the professors in Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 131 relationship to the managers who were not aware of the content of the experimental group treatment. In total this study indicates that working executives can evaluate student papers describing managerial work consistently. it also indicates that students who participate in The Looking Glass will obtain improved knowledge about managerial work. This allows a conclusion that the exercise is useful in management education at the university level. The expectation is that improved realistic knowledge will allow students to better evaluate their potential interest in the field of management and to be more realistically prepared for acting in the managerial environment, which awaits thee. This study is limited in that it concerns only a sample population of students in two beginning management classes at a southern university. The study is relatively small and is limited to results of three executive evaluators, since the faculty evaluators were obviously biased in their scoring. Further study should include a wider range of executives and should include academic evaluators who are not directly involved in the simulation activity. The results, which indicate learning for the experimental group, are based on a typical college-age population, and thus are limited to thee. The results of this study cannot be projected to other groups, in particular, groups of people who are already in the work or managerial environment. An additional consideration is the method of evaluation. Although the three judges showed remarkable consistency, this needs to be demonstrated over a wide range of evaluators. Perhaps the results of a delphi or group consensus technique eight be used to evaluate the student papers rather than evaluators acting independently. Other factors involve the The Looking Class exercise itself. This is an expensive educational intervention f or a class in a business college. It is expensive both in terms of continuing procurement costs for the exercise, class/instructional time, and motivation of instructors. These costs preclude its use in many similar educational environments. In addition, The Looking Class is a very sophisticated large group exercise. This means the results of this study cannot be directly related to other more commonly used small group experiential exercises in management classes. CONCLUSIONS The purpose of this study was to determine whether the simulation The Looking Glass used in a management fundamentals class would actually Improve the students’ knowledge of a manager’s job. Two demographically similar classes at a southern university were used as experimental and control groups. Examination of the data indicates that the executive evaluators found significant improvement in the knowledge of students about the managerial working environment in the experimental group when compared to the control group. It appears that The Looking Glass is an appropriate, though costly, exercise for management education in the university environment. REFERENCES Anderson, Philip H. and Layton, Leigh. (1988). Assessing Student Performance on a Business Simulation Exercise. Proceedings of the Fifteenth Annual Conference of the Association for Business Simulation and Experiential Learning, 15, 24-245. Brenenstuhl, Daniel C. (1975). An Experiential Study of Performance in a Basic Management Course. Proceedings of the Second National ABSEL Conference, 2, 83-91. Gentry, James V., Tice, Thomas F., Robertson, Charles J., Deloitte, Basking and Sells, Gentry, Martha J. (1984). Simulation Gaping as Means of Researching Substantive Issues: Another Look. Proceedings of the 11th Annual Conference of the Association of Business Simulation and Experiential Learning. Honolulu, Hawaii. 11, 1 -4. Gosenpud, Jerry. (1988). Evaluation of Experiential Learning. Proceedings of the Fifteenth Annual Conference of the Association for Business Simulation and Experiential Learning. 15, 276. Johnson, David W. and Johnson, Frank P. (1982). Joining Together Group Theory and Group Skills. Englewood Cliffs, New Jersey: Prentice-Hall, Inc. Lombardo, Michael, McCall, Morgan W., Jr., DeVries, David L. (1983). Looking Glass (Administrators Guide). Center for Creative Leadership: Greensboro, North Carolina. McCall, Morgan V., Jr. and Lombardo, Michael. (1979). Looking Glass, The First Three Years. (Technical Report No. 13). Greensboro, North Carolina: Center for Creative Leadership. Mintzberg. Henry. (1975). “The Manager’s Job: Folklore and Fact,” Harvard Business Review, 53 (July-August); 49. Parasuraman, A. (1980). Evaluation of Simulation Games: A Critical Look at Past Efforts and Future Needs. Proceedings of the 7th Annual Conference of the Association for Business Simulation and Experiential Learning. Dallas Texas. 7, 192-194. Wagner, Richard K. and Sternberg. Robert J. (1987). Tacit Knowledge in Managerial Success, Journal of Business and Psychology, I No. 4, Summer, 1987. Table of Contents Volume 17, 1990 The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - Revisited The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise Use and Effectiveness of an Analogy-Based Expert System Suggestions for Computerized Business Authors Dealing with Power: An Experiential Exercise Using Movie and Personal Diary Analysis Techniques A Model for Developing Student Skills and Assessing Outcomes Through Outdoor Training Computer-Aided Exercises Versus Workbook Exercises as Learning Facilitator in the Principles of Marketing Course An Exposition of Guilford's Si Model as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education Formal Planning and Simulation Team Performance: A Cross Sectional Approach Cases: Real Organizations in Real Time in the Classroom An Empirical Investigation of the Internal Validity of Marketing Simulation Game An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Game in a Principles of Marketing Course Factors Affecting Effective Teaching of Strategic Planning: Some Preliminary Evidence An Experiential Exercise for Learning About the Relationship Between Organizational Form & the Project Management Process Accounting Communication Skills can be Taught in the Auditing Course Modeling Cost Functions in Computerized Business Simulation: An Application of Duality Theory and Sheppard's Lemma A Life Cycle Analysis of Decision Making for a Strategic Management Team What's the Problem? A Dynamic Model for Teaching Problem Solving Skills Experientially International Currency Fluctuations: Money$im, A Simulation Superstores: A Specialized Retailing Simulation Within a Specialized Marketing Curriculum Factors Affecting Student Perceptions of Learning in a Business Policy Game VC + EL = VL The Name Game: An Experiential Exercise in Intergroup Relations The Effects of Experiential Accounting Work Experience on Student Performance in Intermediate Accounting Courses The Results of Using the Experiential Activity Group Performance Evaluation in a Business Policy Setting Using a Legal Database to Describe the Legal Environment of Marketing (and Business) Matching Environmental Uncertainty and Organizational Configuration An Instructional Computer Simulation of Tampering in QC Executive Evaluation of Student Learning in the Looking Glass Simulation Group Personality Composition and Total Enterprise Simulation Performance An Expert System for Selecting Analytical Techniques for Analyzing Marketing Research Data Effects of Cognitive Styles on Responses in an In-Basket Simulation A Psychometric Analysis of Kolb's Revised Learning Style-Inventory Selecting and Developing Experiential Exercises Using Movies Application of a Real-World Strategic Management Model in the Classroom Demand Equations which Include Product Attributes Consumption as the Objective in Computer-Scored Total Enterprise Simulations The Effects of Decision Format and Evaluation on Simulation Performance, Decision Time, and Team Cohesion The Effects of Computer Related Assignments on Student Performance in Business Administration The Money Game: A Dynamic Simulation Including Random Shocks for Money and Banking Courses Methods for Evaluating Performance on Business Simulations: A Survey The Effects of Synergogy on the Policy Course: Significant Improvements in Student Learning and Teacher Evaluation Conditions and Outcomes of Trust in a Two-Person Bargaining Exercise Bankgame Enhancing Computer Business Simulation with the Use of VGA Graphics An Experiential Approach to Entrepreneurship An Advanced Simulation Method (ASM) for Multiple Objective Problems The Influence of Experiential learning Techniques on Student Recognition of Non-Primary Learning Styles Negotiating Mergers and Acquisitions: A Cocktail Napkin Approach Identification of Unintended Effects in Experiential Laboratory Exercises An Experimental Comparison of Paper and Pencil and Computer Aided Decision Support Tools Porting a Simulation from the IBM World to the Macintosh World A Transaction Cost Analysis of Experiential Learning The Development of Experiential Exercises for Courses in Entrepreneurship and Small Business Management The Assessment Center as and Experiential Classroom Exercise Pricing Strategy Algorithms for Playing Business Simulations Organizational Structures for International Operations: An Experiential Activity Simulation Emphasis in the Business School Capstone Course An Integrated Approach to Computerizing the Business Curriculum Cross-Cultural Business Negotiations Exercise Organizational Socialization and Gender Differences in Students at Work Understanding Student Work Experience: A Content-Analytic Approach Introducing Executive MBA Programs with Management Games An Analysis of Improvement in Business Decision Outcome with Sequential Use of Two Simulation Games Teaching Business Policy Utilizing Mass Lecture and Individual Case Labs Potholes Along the Road to Evaluating Learning Outcomes: The Case of Outdoor Management Training Experiential Learning for Interior Design Students: Using CADD, Lotus 1-2-3, and Wordperfect Cognitive Learning Using a Computer-Based, Qualitative Interactive Business Simulation Sex Discrimination: Does the Woman get the Job or Does the Best Man Win? A Search for Visual Aids to Support Experiential Learning Through the 1990's An Experimental Analysis fo the Effectiveness of Student Role-Playing in Sales Training How to Have Students Learn from their Term Projects Self-Evaluation Exercise (SEE): An Assessment of Class Contribution A Study of the Influence of Team Formation on Attitudes and Performance in Management Games A Hardware Based CIM Simulation Laboratory Model Test Substance Abuse in Organizations Micro Computer Training Models Teaching Forecasting, Cash Budgeting and Inventory Model Building Using SBTools A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes An Adjunct Writing Instruction Assistant, The Computer; With an Illustration Classroom Software for ABC Analysis Utilizing Information Processing Technology to Enhance the Business Policy Simulation Experience Progressive Cases Realistic Job Previews Vs. Traditional Job Previews: Experiencing the Differences and Understanding the Consequences Investment Analysis Using the Pragmatic Multiplier Approach A Computer Simulation Interface for Competitive and Firm Analysis Self-Assessment of Ethical Decision Making Predispositions Preparing Managers for Overseas Assignments An Inquiry into Japanese Marketing: Workshop on Teaching Japanese Marketing The Performance Appraisal Feedback Interview: A Role Play for Human Resources Management Teaching Counselor Selling techniques Using Experiential Techniques to Teach International Topics International Management Simulation Gaming: Current Status and Future Developments A Realism Comparison of Simulation Technologies/Methodologies A Time-Efficient Game to Illustrate Concepts Taught in Management Courses and Management Development Programs Teaching the Management of Technology The Concept of Face and the Applicability of Experiential Exercises in an Oriental Culture's