EVALUATION SYSTEMS IN SIMULATION--AN EXPLORATION Computer Simulation and Learning Theory, Volume 3, 1976 281 EVALUATION SYSTEMS IN SIMULATION--AN EXPLORATION H. William Vroman Tennessee Technological University Thomas Urban University of Iowa Introduction Ralph I. Day (Kenderdine, Keys, 1975) noted in his summary comments to an ABSEL Conference that the emphasis in gaming and simulation was implementation. The concern was with the effectiveness of existing simulations rather than new simulations. A major factor was the growing complexity of simulative environments. According to Day, the problem of evaluation was one of the major topics. The problem of evaluation, or alternatively, feedback, assessment, or reinforcement, is certainly prevalent in education in general and in simulation specifically. This is exemplified in the recent spate of activity comparing the effectiveness of case, lecture, and simulation (Wolfe, 1975; Carroll, et al., 1972). Reinforcement Models Reinforcement models are alternate models to consider when attempting to place assessment or evaluation into its proper perspective. Keys’ (1975) model is developed from the work done by Piaget. Learning in his model is defined as a resultant of the interaction of content, experience, and feed-. back. Content encompasses principles, ideas, concepts, and other substantive notions. Experience is the practice of the relevant substance. Feedback occurs when information is made known to the participants relevant to the experience. The quality of the feedback is measured against a criterion of an ideal or a comparison with other participant’s scores. The A.I.R. model was the focus of an article by Urban and Vroman (1975). Assessment, intervention, and reinforcement are the three main variables. Assessment is used in a somewhat different sense than suggested above, in that, the assessment is related to the development of the principles or concepts necessary in the learning process. Evolving from the assessment process is the selection of intervention techniques. Intervention is a more specific, trainer-oriented concept than the Keys concept of experience and includes the myriad social-psychological variables that are always unleashed in a learning environment. Reinforcement occurs when the participant has knowledge of results. An integrative model is elaborated in Figure 1. The variable on the left is the selection of content and pedagogy. Selection of content may determine pedagogy, but the opposite may also occur. A pedagogy may determine the components of the content. The selection of content and pedagogy yields the action or experience. The final stage is the assessment and reinforcement variable. Reinforcement can occur many times during the action, whereas assessment is the final assessment based on the results of the game. Computer Simulation and Learning Theory, Volume 3, 1976 282 Figure 2 adds another dimension to the previous model. In this figure, the experiential and lecture techniques are aligned. The next stage features the relative amount of student involvement in the process according to the intervention technique. Notice that in the experiential exercise, student involvement occurs during the entire portion of the action, whereas, during the lecture, the actual involvement is often right before the assessment time. It is interesting to note that reinforcement is related to involvement. Therefore, the more the student is involved, the more reinforcement can be more easily built into the experiential versus the lecture method. In addition, the evaluation takes on a different tone. The lecture course turns out to be a computational exercise with true/false tests or multiple choice tests. In simulation exercises, the final evaluation is clearly more judgmental (Vroman, 1965). Experiential exercises vary in their complexity. The more complex the game, the closer the experience is to reality. Although in some cases, the impression given by the administrator is that the cause/effect relations are certain and, consequently, the assessment based on a final product such as a profit or a written annual report. The simulation can build in a number of reinforcement points with the evaluation then based on behaviors that are newly learned. The planning process and other process variables can be reinforced and rewarded during the action time between t2 and t5. As a matter of INTEGRATIVE MODEL FIGURE 1 FIGURE 2 Computer Simulation and Learning Theory, Volume 3, 1976 283 fact, these process results can be emphasized and the final result dc-emphasized, which might ~be considered more appropriate. Building Reinforcement into a Simulation Urban and Klatt (1974) built feedback or reinforcement into their model. The model is called “KUBSIM,”(1975). This is an industrial relations game focusing on contract negotiation. The experience comes from the learner assuming the role of a union or management member of a negotiating team. They undergo a multi-faceted performance appraisal. In the game, each student member of the bargaining team chooses a role. The bargaining team, either union or management, gets together for a pre-negotiation bargaining session. In this session, they enumerate the items they will demand, the priority of these items, the initial and lowest acceptable demands, expectations, and justification for their actions. Interestingly, their particular role includes some directions at variance with other members of their own team which illustrates the Walton and McKersie notion of intraorganization bargaining (1965). Once the bargaining position is outlined, the team has provided a standard for post-negotiation evaluation. During the actual negotiations, the teams are paired to negotiate items relating to wages, hours, and working conditions. There is the usual comparison with the pre-negotiation positions while the bargaining is going on. In the post-negotiation period, each team writes up an appraisal of their opponent’s strengths and weaknesses. They point out the successful strategies and the non-successful strategies that were used as well as suggesting improvements. This is an involvement in the appraisal process not available in many simulations. In addition, this is an input to the final evaluation by the instructor. Another appraisal is a post-negotiation audit by a non-opponent team. The audit includes the following ten items: 1. quantitative costs of the contract; 2. quantitative (non-financial) costs of the contract; 3. comparison of post-negotiation results with pre-negotiation standards; 4. comparison of expectations with performance; 5. evaluation of expectations; 6. implementation of roles; 7. strong/weak points of the contract; 8. likely sources of grievances; 9. probable issues for re-negotiation; and 10. provision in the contract for future negotiations. This report is given to the team members as a written report and also presented verbally to the class which causes discussion and further evaluation by Computer Simulation and Learning Theory, Volume 3, 1976 284 the class members. The written opponent report and non-opponent audit are given to the instructor as the basis for final appraisal and evaluation of the team’s performance. A final addition to the performance appraisal is an individual component. This takes the form of a peer appraisal where each member of the team evaluates his own performance as well as the other members. The instructor now has a large number of sources for a final evaluation. The appraisals whereby people intimately involved in the process, the opponent report, and the peer appraisal, as well as an objective audit by a non-opponent team, and then the final component is the instructor’s overall success of the simulation. An interesting difference in this simulation is that behaviors can be rewarded because of the highly professional way that the members went through the process and still didn’t meet their pre-negotiation objectives as well as the opponent written report where strategies are evaluated. Summary A most important variable in a system of learning is evaluation. Two models are introduced and an integration model proposed. The Keys’ model encompassed the variables content, experience, and feedback. The A.I.R. model uses similar variables in assessment, intervention, and reinforcement. The integrative model proposed selection of content-pedagogy, action, and evaluation as the main variables. The Klatt-Urban simulation, KUBSIM, builds a number of reinforcement points into the action of the simulation. With the several evaluations, the instructor no longer can grade. computationally, but instead grades judgmentally. Computer Simulation and Learning Theory, Volume 3, 1976 285 REFERENCES Carroll, S. J., Paine, F. T. and J. J. Ivancevich, “The Relative Effectiveness of Training Methods--Expert Opinion and Research,” Personnel Psychology, 25 (1972), pp. 295-509. Kenderdine, James, and Bernard Keys, Simulations, Games and Experiential Learning Techniques. Norman, Oklahoma: Center for Economic and Management Research, 1975. Keys, Bernard, “The 9-9-9 Manager of Learning,” paper presented at the National Meetings, Academy of Management. New Orleans, 1975. Klatt, L. A. and T. F. Urban, KUBSIM: A Simulation in Collective Bargaining. Columbus, Ohio: Grid, Inc., 1975. Urban, T. F. and L. A. Klatt, “Multi-dimensional Evolution of Non-Quantifiable Criteria in Simulation,” paper presented at the National Conference, American Institute of Decision Sciences. Atlanta, Georgia, 1974. Vroman, H. W., “Differentiating MBO-Appraisal Systems: A Contingency View.” Journal of Business Research, 3, (January, 1975) pp. 53-60. Walton, R. E. and R. B. McKersie, A Behavioral Theory of Labor Negotiations. New York: McGraw-Hill, 1965. Wolfe, J., “A Comparative Evaluation of the Experiential Approach as a Business Policy Learning Environment,” Academy of Management Journal, 18, (1975) pp. 442-451.