Evaluation Model of the Global Performance of a Management Simulation for the Academic Environment EVALUATION MODEL OF THE GLOBAL PERFORMANCE OF A MANAGEMENT SIMULATION FOR THE ACADEMIC ENVIRONMENT Ricardo R. S. Bernard Universidade Federal de Santa Catarina bernard@cse.ufsc.br Moisés Pacheco de Souza Universidade Federal de Santa Catarina mpsouza1980@yahoo.com.br Maurício V. L. Lyrio Universidade Federal de Santa Catarina mauriciovll@gmail.com ABSTRACT This paper proposes and tests an evaluation model of performance in an exercise of management simulation taking into account the indicators identified by the ones involved in the process, i.e., professor and students. For the construction of the model the Multiple Criteria Decision Aid (MCDA) method was used in a management simulation course. Seventeen (17) criteria were identified in order to be used for the evaluation of the performance of the simulation. The methodology demonstrated what would be considered in such criteria and their relative importance. Once the evaluation model was created, it was tested in the same class that conceived it. As a result, the application of the exercise of management simulation pointed to a global performance of 88 points out of 100, the number considered by the professor as a good score. It involved not only traditional evaluation criteria of students and teams, but also the characteristics of the professor, the students, the simulator and the simulated environment. Key-words: Management simulation; Business game; Performance evaluation, Multiple Criteria Decision Aid, MCDA. INTRODUCTION The main goal of utilizing the management simulation in the academic environment is to develop students’ knowledge as regards the dynamic business environment as well as the improvement of the skills and attitudes of its participants. As defined by Keys & Wolfe (1990:1), “management games are used to create experimental environments within which learning and behavioral changes can occur and in which managerial behavior can be observed”. Many perspectives have been studied to evaluate the performance in exercises of management simulation. This paper proposes a new perspective of performance evaluation focusing on the global performance of a class in the management simulation exercise. Such a way of evaluation reveals both strong and weak points of an exercise of management simulation. In order to obtain the global performance, the authors developed an evaluation model of performance of a class in the management simulation exercise by making use of the Multiple Criteria Decision Aid – Constructivist (MCDA-C) methodology as the instrument of intervention. Such a model comprises both the perceptions of students and the professor in identifying the criteria to be evaluated. This methodology attempts to consider the perceptions and values of those involved in the process so as to identify the elements to be considered for the evaluation by developing an adequate model for the specific situation under analysis. The goal of this paper is therefore to construct and test an evaluation model of performance of a class in an exercise of management simulation which involves the perceptions of both the students and the professor, thus allowing a more adequate way of performance evaluation as regards the criteria they consider important. EVALUATION IN MANAGEMENT SIMULATION The evaluation of an exercise of management simulation can be carried out under several views. One of the most investigated views is the learning that the management simulation provides to its participants. At the beginning, the learning was assumed to be positively related to simulated company performance (Teach, 2007). But, this assumption was not supported in many studies (Anderson & Lawton, 1990; Anderson & Lawton, 1997; Teach, 1990; Washbush & Gosen, 2001). However, many rigorous Developments in Business Simulation and Experiential Learning, Volume 35, 2008 252 mailto:bernard@cse.ufsc.br mailto:mpsouza1980@yahoo.com.br mailto:mauriciovll@gmail.com studies have proved that management simulation does provide some learning, as reviewed by Gosenpud (1990). What is in discussion, as stressed by Faria (2001) is ‘What is learned?’, ‘What type of learning occurs?’ and ‘How does learning occur?’ As a result of one overview of researches on learning of business simulation until the late nineties, the author categorized six periods, as follows (Faria, 2001:105): (a) Many studies identifying specific issues learned through business games (1974 to 1976); (b) Extension of basic learning studies from students to business executives and simulation administrators (late 1970s and early 1980s); (c) Overviews of learning studies (mid-1980s); (d) Agreement that some form of learning takes place with the use of business simulation/games (late 1980s); (e) A shift in research from what is learned to how learning takes place (early 1990s); and (f) Attempts to design studies that will prove cognitive and behavioral learning occur through the use of business games (late 1990s). In a complementary view, Schumann et al. (2001) suggest a framework for evaluating simulations as educational tools. For them, learning is just one aspect to be evaluated (level 2). Other aspects would include the reactions the participants show towards the experience (level 1), the level of change of behavior (level 3), and finally, the benefits they may provide later to their workplaces (level 4). The evaluations of the reactions towards the experience are generally measured through variables such as satisfaction and motivation, two factors that have been investigated by many authors. The assumption behind many of such investigations is that these factors may be considered as variables that precede learning. Yet the levels of change of behavior and later benefits, although deemed easy to be analyzed, are difficult to be measured as they normally require more complex designs and involve longitudinal studies; in addition, the variables under observation are susceptible to have the influence of several exogenous factors. More recently, research is being conducted to verify if the way participants react to the simulated performance can affect their learning. For example, if students with a learning orientation react more favorably to a negative outcome in simulation games than students with a performance orientation. Preliminary findings have presented inconclusive results (Gentry et al., 2007). It should be also pointed out that the role played by the professor must also be taken into consideration as, according to Keys & Wolfe (1990:314), the way he/she manages a simulation is probably the most important factor for the success of an application. In spite of such evidence, research on the impact of the professor’s variables upon the performance of a simulation exercise has not been found in the literature. This paper is based on the level 1 of the framework presented by Schumann et al. (2001) for the evaluation of a management simulation, involving not only traditional evaluation criteria of students and teams, but also the characteristics of the professor, the students, the simulator and the simulated environment. It must be highlighted that the variables chosen for the evaluation of an exercise of management simulation were one of the results of the research, according to the perception of those involved in the process. MULTIPLE CRITERIA DECISION AID – CONSTRUCTIVIST (MCDA-C) METHODOLOGY The Multiple Criteria Decision Aid – Constructivist (MCDA-C) is one of the segments of the multicriteria methodologies, a research area which is considered an evolution of the Operational Research. The multicriteria approach may be considered as having two main segments: on the one side, the MCDM proposes to develop a mathematical model which allows the discovery of “that” optimum solution which is believed to be pre-existent, notwithstanding the individuals involved. On the other side, the MCDA attempts to help modeling the decision context departing from the consideration of convictions and values of the individuals involved by seeking to construct a model which is founded on the decisions that favor what is believed to be most adequate (Roy, 1990). The position related to the decision situation – while the MCDM seeks an optimum solution, the MCDA seeks an adequate solution – may be considered the main difference between these two currents of thought. The process of support to decision developed by the MCDA-C is permeated by Piaget’s constructivist view, according to which knowledge is the result of some kind of interaction between the subjective and the objective elements, i.e., interaction between an active individual looking for an adaptation to an object – an engagement which results in a representation that is objectively valid and subjectively significant (Landry, 1995:326). CONSTRUCTION OF THE MODEL The group chosen for the construction of the model was a class of 32 undergraduate students who were taking “Business Game II”, a course of the last period of Accountancy in the Federal University of Santa Catarina (UFSC), Brazil. The criterion for the selection of the group was intentional, i.e., the class had already taken the course “Business Game I” and the students had already had, therefore, a previous experience with management simulations as well as with a system of method evaluation. Thus, students were expected to provide more criteria to be taken into account by the model. A random selection was performed to choose one student of each team. As a result, 8 students were chosen to help in the construction of the model. As soon as the model was devised, all the 32 students have also received a questionnaire by e-mail to provide the necessary information to test the model. The Developments in Business Simulation and Experiential Learning, Volume 35, 2008 253 questionnaire had a 25% response rate. Detailed information about the entire construction of the model is provided next. For the construction of the model the MCDA-C methodology was employed in three stages, as suggested by Ensslin (2002). Stage I – Structuring: it consists of understanding and ordering the decision context (creation of the decision tree and attributes). Stage II – Evaluation: it consists of developing local cardinal scales for the attributes created and identifying the substitution rates by informing the relative importance of each attribute for the global result of the model. In this stage the application of the model is also carried out. Stage III – Making Recommendations: it consists of suggesting potential actions with the goal of improving students’ performance in the exercise of management simulation. Stage I – Structuring: The structuring stage was divided into two phases: (a) identifying the actors involved in the decision context and (b) structuring such a context. (a) The actors were divided into two categories: • Those acted upon (students that were not interviewed) – with no power of decision. They simply undergo the consequences of the decision established by the interveners. • Interveners – these have the power of decision as they directly act in the decisions taken. The interveners are divided into decision-maker (the professor), demanders (students who were interviewed and who represent the teams), and facilitators (responsible for the creation, data gathering and testing of the model). The facilitators are not totally active. However, they provide support to the decision and suggest recommendations. (b) The structuring of the decision process was divided into four steps: • Step 1: Definition of the label of the problem. • Step 2: Survey of the Primary Evaluation Elements (PEEs). • Step 3: Construction of the point-of-view arbor. • Step 4: Construction of the attributes. Table 1 – Primary Evaluation Elements (PEEs) from the professor’s point of view PROFESSOR Code PEE Code PEE 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Access to the website Team members affinity Competitor analyses Analyses of the simulated results Learning Simulation learning Class attendance Delays Managerial capabilities Scenario Complexity Specific managerial concepts Concepts of the company’s functions Managerial concept Academic concepts Competition Strong competition Knowledge Company knowledge Managerial knowledge Knowledge consolidation Context of the simulation Academic performance Managerial performance Demotivation Knowledge initiation Didactic Team assignments Teaching Understanding of the simulator 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 Evolution Experience Market experience Familiarity with the simulation model Feedback Presence Managerial indicator Integration of the functional decisions Interaction Autocratic leader Democratic leader Motivation Practical level Theoretical level Simulation objectives Participation Experience with the simulation model Presence in the classroom Affinity problems with the professor Personal problems Professor’s desired characteristics to use the method Students’ interest in checking the simulated results Professor-students relationship Managerial results Theory Teamwork Macroeconomic variables Market vision Practical experience Developments in Business Simulation and Experiential Learning, Volume 35, 2008 254 1 Ac ad em ic pe rfo rm an ce 1.1 Pr ofe ss or 1.2 Stu de nt 1.2 .2 M oti va tio n 1.2 .3 Stu de nt s m an ag em en t ex pe rie nc e To ev alu ate stu de nts ’ a ve ra ge gr ad es in wr itte n wo rks 1.2 .2. 1 Cl as s a tte nd an ce 1.2 .2. 2 Ac ce ss to th e we bs ite To ev alu ate stu de nts ’ a ve ra ge att en da nc e To ev alu ate stu de nt’ s a ve ra ge win do w tim e be twe en th e p os ted re su lts an d t he ac ce ss to th e we bs ite To ev alu ate stu de nts ’ a ve ra ge ex pe rie nc e i n co m pa ny m an ag em en t 1.1 .1 Sim ula tio n ob jec tiv es 1.1 .5 Ba ck gr ou nd / E du ca tio n To ev alu ate th e ob jec tiv es of th e sim ula tio n ex er cis e 1.1 .2 Ex pe rie nc e w ith th e m eth od 1.1 .3 Ex pe rie nc e w ith the m od el To ev alu ate th e pr ofe ss or ’s ex pe rie nc e w ith the m eth od To ev alu ate th e pr ofe ss or ’s ex pe rie nc e w ith the m od el us ed in the si m ula tio n ex er cis e 1.1 .4 Pr ofe ss or s m an ag em en t ex pe rie nc e To ev alu ate th e pr ofe ss or ’s ye ar s o f ex pe rie nc e i n co m pa ny m an ag em en t. To ev alu ate th e pr ofe ss or ’s ba ck gr ou nd /ed uc ati on 1.2 .1 W ritt en wo rks 60 % 30 % 70 % 20 % 3% 40 % 30 % 7% 40 % 50 % 10 % 50 % 50 % 2 M an ag er ial Pe rfo rm an ce 2.1 Sim ula ted en vir on m en t 2.3 Te am 2.3 .3 Le ad er sh ip To ev alu ate th e nu m be r o f te am s tha t h ad re lat ion sh ip pr ob lem s w ith th e pr ofe ss or of th e dis cip lin e 2.3 .2 Stu de nt- stu de nt re lat ion sh ip To ev alu ate th e nu m be r o f te am s tha t h ad re lat ion sh ip pr ob lem s i ns ide th e tea m To ev alu ate th e nu m be r o f te am s wit h a n a uth or ita ria n lea de r o r w ith ou t a lea de r 2.1 .1 Co m ple xit y 2.2 .2 Co m pa ny Ind ica tor s To ev alu ate th e nu m be r o f d ec isi on va ria ble s e xis ten t in the si m ula tio n m od el 2.1 .2 M ac ro -e co no mi c ind ice s 2.1 .3 Co m pe titio n To ev alu ate th e co m bin ati on of m ac ro - ec on om ic ind ice s u se d in the si m ula tio n b y tak ing in to ac co un t: hig h i nfl ati on ra te; lo w ec on om ic gr ow th; hi gh pa rtic ipa tio n o f im po rte d p ro du cts ; h igh re ad jus tm en t o f su pp lie rs; hi gh in ter es t ra tes To ev alu ate th e m ar ke t s ha re of the si m ula ted firm s 2.2 .1 De cis ion qu ali ty To ev alu ate th e nu m be r o f co m pa nie s t ha t ra tio na lly m ad e u se of the in for m ati on wit h t he su pp or t o f ca lcu lat or s, sp re ad sh ee ts an d/o r m ate ria l n ot re qu ire d by th e p ro fes so r. To ev alu ate the av er ag e gr ow th of the ne t p ro fit of the co m pa nie s i n the si m ula tio n ex er cis e i n co m pa ris on to the in itia l v alu e 2.3 .1 Pr ofe ss or /st ud en t re lat ion sh ip 2.2 Sim ula ted C om pa ny 40 % 50 % 30 % 20 % 20 % 50 % 30 % 50 % 50 % 20 % 40 % 40 % Pe rfo rm an ce E va lua tio n i n a Ma na ge me nt Sim ula tio n Figure 1 – Constructed model of global performance of a management simulation Developments in Business Simulation and Experiential Learning, Volume 35, 2008 255 Step 1 – Definition of the label of the problem: The label is the statement of the problem. It must carry the focus of the work, the goal to be achieved and not to leave any traces of doubt. In this paper, the label of the model was defined as Construction of an Evaluation Model of Performance for a Management Simulation Class. Step 2 – Survey of the PEEs: After defining the decision context and the label of the problem, the structuring of the model itself is started. For such, in the first place the PEEs must be surveyed, as they are the first concerns that come to the decision-maker’s mind as regards the decision situation. The PEEs are surveyed by means of the brainstorm technique in which the decision-maker is invited to discuss about the situation by surveying the concerns that come to his/her mind as regards the problem, without any kind of limitation. After this interaction, sorting is carried out not considering the redundant PEEs or the ones that are considered irrelevant. For this specific paper, the PEEs were surveyed by means of 8 (eight) semi-structured interviews representing one student for each simulated company and the professor of the management simulation course. The questions raised were the starting point for the discussion instead of a script strictly followed so as to avoid the heading of the answers given by the decision-makers. By means of such interviews 99 PEEs related to the performance in a management simulation exercise were obtained, broken down as follows: 59 PEEs were extracted from the interview with the professor, whereas 40 were extracted from the interviews with the students. The 99 PEEs surveyed from the interviews were grouped according to the affinity of ideas, as described by Eden (1988), which resulted in 24 PEEs. Table 1 and Table 2 present all the PEEs obtained through the interviews with the professor and with the students respectively, while Table 3 shows the final PEEs. Step 3 - Construction of the point-of-view tree: The models based on the MCDA-C are normally organized in the form of an decision tree: the label of the problem is placed at the highest level, then the areas of interest come right below it, followed by the Fundamental Points of View (FPVs), and finally, if necessary, the Elementary Points of View (EPVs) are displayed. The EPVs are unfolded until they come to a susceptible level of measurement. The 24 PEEs were reorganized in a hierarchical way so as to facilitate the understanding, as presented in Figure 1. Step 4 – Construction of the attributes: Once the decision tree has been constructed, the next step of the structuring stage consists of the construction of the attributes, which are the tools used for measuring and evaluating the performance of the potential actions (in the case, the potential action will be the performance of the class in exercising the management simulation). Table 4 presents all the attributes created for the model with their respective value functions. The attribute, according to Kenney & Raiffa (1993:32) “provides a scale for measuring the degree to which its respective objective is met”. Once the phase of attributes’ construction is finished, the stage of the model’s structuring is concluded. Stage II – Evaluation: The evaluation stage starts with the construction of local cardinal scales for the attributes’ levels. This process makes use of the Macbeth-Scores software (Bana and Costa, Vasnick, 1997), in which the Table 2 – Primary Evaluation Elements (PEEs) from the students’ point of view STUDENTS Code PEE Code PEE 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 Market environment Learning Discussion Goal achievements Autocratic leader Market characteristics Coherence Competition Added knowledge Initial knowledge Stock market value Erroneous decisions Defense of opinions Defense of ideas Understanding Market understanding Strategy Experience Class attendance Basic information 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Justification of the decisions Leadership Earning Motivation Economic concepts Objectives Divergence of ideas Planning Professor behavior Consequences of the decisions Mathematic formulas of the model Respect to the student’s viewpoint Respect to the team member Theory Work in teams Teamwork Strategy Professional life Market vision Systemic vision Developments in Business Simulation and Experiential Learning, Volume 35, 2008 256 Table 3 – Final Primary Evaluation Elements (PEEs) Final PEEs Access to the website (1) Simulated environment (10, 22, 60, 65) Student¹ Complexity (11) Competition (3, 16, 17, 67) Academic performance (23) Managerial performance (9, 24, 54) Simulated company ² Students’ management experience (77, 97, 98, 99) Professor’s management experience (13, 19, 20, 32, 33, 43, 58 59) Experience with the model (30, 34, 47, 51) Background/Education (12, 14, 15, 18, 21, 27, 29, 44, 55, 84) Class attendance (7, 8, 36, 48, 78) Macroeconomic indices (57, 75) Leadership (40, 41, 64, 81, 86) Motivation (25, 42, 46, 52 83) Simulation objectives (5, 6, 26, 31, 45, 61, 68, 69, 74, 85) Professor³ Company indicators (70, 82) Decision quality (35, 43, 63, 66, 71, 76, 80, 87, 89, 90, 96) Student-student relationship (28, 39, 56, 62, 72, 73, 91, 92, 94, 95) Professor-student relationship (49, 50, 53, 88) Written works (94, 95) ¹ Including the PEEs Written works, Motivation, Class attendance, Access to the website and Student´s management experience. ² Including the PEEs Decision quality and Company indicators. ³ Including the PEEs Professor´s management experience, Experience with the simulator, Background/Education and Simulation objectives. Table 4: Attributes and value functions for all the Elementary Points of View (EPV) Attribute 1.1.1: Simulation objectives Objective: To evaluate the objectives of the simulation exercise. Impact Levels Reference Levels Description Value Function L5 The management simulation course had specific pedagogical goals. The professor was clear about these goals. The goals were achieved. Goals not initially defined were also achieved. 150 L4 GOOD The management simulation course had specific pedagogical goals. The professor was clear about these goals. The goals were achieved. 100 L3 NEUTRAL The management simulation course had specific pedagogical goals. The professor was clear about these goals. However, the goals were not achieved. 0 L2 The management simulation course had specific pedagogical goals. However, the professor was not clear about these goals and the students did not achieve them. -150 L1 The management simulation course had not specific pedagogical goals. The professor only run the simulation and the students were focused only in achieving the best simulated performance results. -175 Attribute 1.1.2: Experience with the method Objective: To evaluate the professor’s experience with the method. Impact Levels Reference Levels Description Value Function L5 More than 2 administrations 127 L4 GOOD 2 administrations 100 L3 1 administration 55 L2 NEUTRAL Only experience as participant 0 L1 Without experience -55 Attribute 1.1.3: Experience with the model Objective: To evaluate the professor’s experience with the model used in the simulation exercise. Impact Levels Reference Levels Description Value Function L5 More than 4 administrations 200 L4 3 a 4 administrations 175 L3 GOOD 2 administrations 100 L2 NEUTRAL 1 administration 0 L1 Without experience -125 Developments in Business Simulation and Experiential Learning, Volume 35, 2008 257 Attribute 1.1.4: Professor´s management experience Objective: To evaluate the professor’s years of experience in company management. Impact Levels Reference Levels Description Value Function L5 More than 10 years of experience 160 L4 5 to 10 years of experience 140 L3 GOOD 1 to 5 years of experience 100 L2 NEUTRAL Up to 1 year of experience 0 L1 Without experience -120 Attribute 1.1.5: Background / Education Objective: To evaluate the professor’s background/education. Impact Levels Reference Levels Description Value Function L4 GOOD Undergraduate and graduate degree in business. 100 L3 Undergraduate degree in business or undergraduate degree in other fields, but graduate degree in business. 56 L2 NEUTRAL Undergraduate degree not related to business. 0 L1 No undergraduate degree. -67 Attribute 1.2.1: Written works Objective: To evaluate students’ average grades in written works. Impact Levels Reference Levels Description Value Function L6 More than 9.0 points 125 L5 GOOD 8.0 to 9.0 points 100 L4 6.0 to 7.9 points 50 L3 NEUTRAL 4.0 to 5.9 points 0 L2 1.1 to 3.9 points -50 L1 Up to 1.0 point -75 Attribute 1.2.2.1: Class attendance Objective: To evaluate students’ average attendance. Impact Levels Reference Levels Description Value Function L5 Higher than 90% 125 L4 GOOD 86% to 90% 100 L3 81% to 85% 75 L2 NEUTRAL 76% to 80% 0 L1 Up to 75% -125 Attribute 1.2.2.2: Access to the website Objective: To evaluate student’s average window time between the posted results and the access to the website. Impact Levels Reference Levels Description Value Function L4 All students had accessed the results at the same day that they were posted. 140 L3 GOOD Average window time access was 1 day after the posted results and 2 days before the new decision making process. 100 L2 NEUTRAL Average access window time was 1 day before the new decision making process. 0 L1 Students had never accessed the website. -120 Developments in Business Simulation and Experiential Learning, Volume 35, 2008 258 Attribute 1.2.3: Student´s management experience Objective: To evaluate students’ average experience in company management. Impact Levels Reference Levels Description Value Function L5 More than 10 years of experience 133 L4 5 to 10 years of experience 117 L3 GOOD 1 to 5 years of experience 100 L2 Up to 1 year of experience 67 L1 NEUTRAL Without experience 0 Attribute 2.1.1: Complexity Objective: To evaluate the number of decision variables existent in the simulation model. Impact Levels Reference Levels Description Value Function L4 More than 30 variables 225 L3 GOOD 26 to 30 variables 100 L2 15 to 25 variables 50 L1 NEUTRAL Lower than 15 variables 0 Attribute 2.1.2: Macroeconomic indices Objective: To evaluate the combination of macro-economic indices used in the simulation by taking into account: high inflation rate; low economic growth; high participation of imported products; high readjustment of suppliers; high interest rates. Impact Levels Reference Levels Description Value Function L5 GOOD 1 indicator 100 L4 2 indicators 54 L3 3 indicators 23 L2 NEUTRAL 4 indicators 0 L1 5 indicators -8 Attribute 2.1.3: Competition Objective: To evaluate the market share of the simulated companies. Impact Levels Reference Levels Description Value Function L5 Simulated companies had similar market share. 150 L4 GOOD 4 simulated companies had together more than 50% of the market. 100 L3 3 simulated companies had together more together than 50% of market. 50 L2 NEUTRAL 2 simulated companies had together more together than 50% of market. 0 L1 1 simulated company had more than 50% of market share. -75 Attribute 2.2.1: Decision quality Objective: To evaluate the number of companies that rationally made use of the information with the support of calculators, spreadsheets and/or material not required by the professor. Impact Levels Reference Levels Description Value Function L5 All simulated companies (8) 150 L4 GOOD 5 to 7 simulated companies 100 L3 4 to 5 simulated companies 50 L2 NEUTRAL 1 to 3 simulated companies 0 L1 0 simulated companies -50 Developments in Business Simulation and Experiential Learning, Volume 35, 2008 259 Attribute 2.2.2: Company indicators Objective: To evaluate the average growth of the net profit of the companies in the simulation exercise in comparison to the initial value. Impact Levels Reference Levels Description Value Function L5 Higher than 100% 140 L4 GOOD 51% to 100% 100 L3 26% to 50% 60 L2 NEUTRAL -25% to 25% 0 L1 Lower than - 26%. -60 Attribute 2.3.1: Professor-student relationship Objective: To evaluate the number of teams that had relationship problems with the professor of the discipline. Impact Levels Reference Levels Description Value Function L6 0 teams 200 L5 1 team 150 L4 GOOD 2to 3 teams 100 L3 4 to 5 teams 50 L2 NEUTRAL 6 to 7 teams 0 L1 All teams (8) -50 Attribute 2.3.2: Student-Student relationship Objective: To evaluate the number of teams that had relationship problems inside the team. Impact Levels Reference Levels Description Value Function L6 0 teams 200 L5 1 team 150 L4 GOOD 2 to 3 teams 100 L3 4 to 5 teams 50 L2 NEUTRAL 6 to 7 teams 0 L1 All teams (8) -50 Attribute 2.3.3: Leadership Objective: To evaluate the number of teams with an authoritarian leader or without a leader. Impact Levels Reference Levels Description Value Function L6 0 teams 200 L5 1 team 150 L4 GOOD 2 to 3 teams 100 L3 4 to 5 teams 50 L2 NEUTRAL 6 to 7 teams 0 L1 All teams (8) -50 Developments in Business Simulation and Experiential Learning, Volume 35, 2008 260 Figure 2 – Example of one value function generated by the Macbeth-Scores software levels of anchorage for the attributes are defined (Neutral Level and Good Level). The area above the superior limit is considered the level of excellence that is aimed at, whereas the area below the inferior limit is considered inadequate, thus being penalized by the model. Once the anchorage takes place, it is time to establish the differences of attractiveness between the attributes’ levels. For such, it is necessary to create a value function for each attribute by making use of the semantic judgement method through one- by-one comparisons (Bana and Costa, Stewart, Vansnick, 1995), as shown in Figure 2. The next phase of the evaluation consists of identifying the substitution rates that inform the relative importance of each criterion of the model. Upon obtaining the substitution rates of each one of the criteria, it is possible to turn the evaluation value of each criterion into values of a global evaluation. There are several methods for such, as the Trade-off (Bodily, 1985; Von Winterfeldt, Edwards, 1986; Watson & Buede, 1987; Keeney, 1992; Beinat, 1995), the Swing Weights (Bodily, 1985; Von Winterfeldt, Edwards, 1986; Goodwin & Wright, 1991; Keeney, 1992; Beinat, 1995), and the One-to-one comparison (Beinat, 1995; Larichev & Moshkovich, 1997). For this paper the substitution rates were obtained by means of the Swing Weights method, which consists of requesting the decision-maker (the professor) to choose, as of a fictitious action with performance at the Neutral level of impact in all criteria, a criterion in which the action performance improves until it reaches the Good level. Such a leap forward is worth 100 points. Next, the decision-maker is requested to define, among the remaining criteria, which one he/she would like to have a leap from the Neutral level to the Good level, and how much this leap would be worth in relation to the first one; this step is repeated for all other criteria of the model (Ensslin et al., 2001:224-225). As an example, take the establishment of the substitution rates for the sub-EPVs 2.1.1 – complexity, 2.1.2 – macroeconomic indices and 2.1.3 – Competition, in relation to the EPV 2.1 – simulated environment. The decision-maker deemed the first leap should have taken place at the sub-EPV 2.1.2, thus assigning 100 points to it. Next, 60 points were assigned to the sub-EPV 2.1.3 and 40 points to the sub-EPV 2.1.1. At last, it is necessary to equalize such values so that they total 1 by dividing the points related to each criterion by the total of points. This way, the substitution rates are: 2.1.1 – Complexity w1 = 40/200 = 0.20 or 20% 2.1.2 – Macro-economic indices w2 = 100/200 = 0.50 or 50% 2.1.3 – Competition w3 = 60/200 = 0.30 or 30% Once the substitution rates have been replaced, the evaluation model is concluded and has already reached its largest goal – to generate understanding about the decision context – which is taken as important for the performance evaluation of a class in an exercise of management simulation. Nevertheless, it is also an objective to know the global performance of the class in the exercise of management simulation and this leads to the aggregation of the local evaluations (evaluation of the EPVs/criteria). The global evaluation of an action/alternative is calculated by means of the following mathematical equation of additive aggregation: ( ) ( ) ( ) ( ) ( )aVWaVWaVWaVWaV nn *...*** 332211 +++= where: ( ) eglobalvaluaV = ( ) ( ) ( ) =aVaVaV n,..., 21 partial value of the criteria 1, 2, 3, …, n. W1, W2… Wn = substitution rates of the criteria 1, 2, 3… n. n = number of criteria in the model. Stage III – Making Recommendations: In this stage it is suggested the potential actions to improve the performance. The process of making the recommendation actions is carried out based on the attributes whose performances did not meet the decision-makers’ expectations. ANALYSIS AND APPLICATION OF THE MODEL Based on the application of the proposed methodology, it was possible to construct an evaluation model of performance founded on the perceptions of the ones Developments in Business Simulation and Experiential Learning, Volume 35, 2008 261 involved (professor and students that were interviewed) in a subject of management simulation. Departing from the process of the model’s construction, it was possible to identify 17 (seventeen) criteria that should make up the model to be used for evaluating the performance of a management simulation class, as follows: 1.1 – Professor, subdivided into 1.1.1 – simulation objectives, 1.1.2 – experience with the method, 1.1.3 – experience with the simulator, 1.1.4 – professor’s management experience, and 1.1.5 – background/education; 1.2 – Student, subdivided into 1.2.1 – written works; 1.2.2 – motivation (explained by 1.2.2.1 – class attendance and 1.2.2.2 – access to the website), and 1.2.3 – students’ management experience; 2.1 – Simulated environment, subdivided into 2.1.1 – complexity, 2.1.2 – macroeconomic indices and 2.1.3 – competition; 2.2 – Simulated company, subdivided into 2.2.1 – decision quality, and 2.2.2 – company indicators; and, finally, 2.3 – Team, subdivided into 2.3.1 – professor-student relationship, 2.3.2 – student- student relationship, and 2.3.3 – leadership. Figure 1 presents the model constructed in this paper, which shows the 17 (seventeen) criteria as well as the simulated performance profile of the class under investigation. The performance of each criterion was obtained by means of information regarding the simulated environment (simulator’s data), the professor (personal and group’s data), and the students (when the information could not be obtained by the professor). The information collected directly with students was received by means of a questionnaire sent by e-mail (25% of return rate). The questions were concerned to ‘years of managerial experience in real-world companies’, ‘the use of calculators, spreadsheet software and bibliographical references to support the decision making process’, ‘the existence of student-professor relationship problems’, ‘the existence relationship problems inside the team’, and ‘the leadership style of the team-member leader’. Once the information was collected, the global evaluation could take place by means of the additive aggregation method: V(a) = {0.60 * [0.30 * (0.20 * 100 + 0.07 * 127 + 0.03 * 200 + 0.4 * 160 + 0.3* 100)] + [0.70 * ((0.40 * 50 + 0.50 * (0.50 * 75 + 0.50 * 100)) + 0.10 * 67)]} + {0.40* [0.50 * (0.20 * 50 + 0.50 * 100 + 0.30 * 50)] + [0.30 * (0.50 * 100 + 0.50 * 0)] + [0.20 * (0.20 * 200 + 0.40 * 150 + 0.40 * 200)]} = 88 The positive punctuation of 88 was obtained as the result provided by the performance evaluation of a class of management simulation, in a scale from “0” (Neutral Level or Minimum Acceptable) to “100” (Good Level), which characterizes a performance near to the level which is considered to be good by the decision-maker (the professor). However, sheer identification of such a performance profile is not enough to aid the improvement process of students’ performance. Thus, the graphic representation of the performance profile is elucidating in the sense that it allows the visualization of those Elementary Points of View – EPVs (or criteria) responsible for the inadequacy of the performance of the class under investigation. As shown in Figure 3, criteria 1.2.1 – written works, 1.2.2 – attendance, 1.2.3 – students’ management experience, 2.1.1 – complexity, and 2.1.3 – competition are the weak points of the class’s performance. By having the criteria that jeopardize the global performance of the class it is then possible to propose the actions for improvement. As guided during the making of recommendations, the generation process of actions of improvement is carried out based on the attributes. An important aspect of the model is the possibility it offers to verify the specific performances by means of the analysis of the ramifications of the decision tree. After the application of the model, it was possible to verify that the professor, for having experience with the method of management simulation and with the simulator, as well as for having good academic background knowledge and experience in management of real companies, had an excellent performance. His punctuation reached 129 points, which is considered an excellent performance. Yet students got 70 points, mainly because of the criteria “written works”, “attendance” and “students’ management experience”. This analysis allowed to verify that the professor’s performance was above the “good” level (100 points), while students’ performance was below the level considered “good” for the decision-maker (the professor). The global performance of the simulation exercise, on its turn, underwent greater influence of the students’ criteria because they had a heavier weight in the decision tree. CONCLUSION In this paper it was developed and applied a new approach to performance evaluation of an exercise of management simulation founded on the perceptions and values of those involved in the process, i.e., the professor of the course and his/her students, and showed, in an objective and clear way, the performance of the class under analysis. As some perceptions provided by the students could be influenced by the professor knowledge of such information, the students were advised that all information would be only disclosed after the course was finished and anonymously. Thus, the students were free to provide sensitive information without having their grades compromised by the professor’s judgement. Another result obtained was the possibility to compare the different views – of both professor and students – in regard to the evaluation system, as presented in Table 4. The model constructed allows the evaluation not only of the global performance of the class but also the performance of the professor, the students, the simulated environment, the simulated company or the teams, as well as the analysis of the distinct ramifications of the decision tree. The application of the model constructed take place in two different lines: (i) to improve the understanding about the criteria considered important in the evaluation of a class in a management simulation exercise, both from the perspective of the professor and the students involved in the Developments in Business Simulation and Experiential Learning, Volume 35, 2008 262 process, and (ii) to measure the performance of a class on the basis of objective criteria, minimizing the ambiguity of the evaluation process and providing the implementation of improvement actions on the grounds of the criteria in which the class is not on adequate levels. However, the evaluation criteria of the applied model cannot be generalized because it was devised considering the perceptions and values of a specific class. Given such a situation, the model must be calibrated in each future application, taking into account the different perceptions of the professor (decision-maker) and the students (demanders) as regards the criteria to be chosen to evaluate a management simulation course and their relative importance. For example, in the evaluation model suggested, the complexity of the simulator was considered by the decision- maker (the professor) as a positive criterion. At a first glance, this choice contradicts the theory that learning may occur with both simple and complex simulators (Keys & Wolfe, 1990; Feinstein & Cannon, 2002). However, in this particular application, the use of a more complex simulator was important because the goal of the simulation was to give a holistic view of a company’s operation and such a view might not have been obtained if had a simpler simulator been used. This is one of the reasons that ratify the importance of stating that the model suggested is idiosyncratic for a given class. The maximum that may be utilized is the methodology and a suggestion of the criteria employed. As a final comment, it is important to highlight that the proposed evaluation model is an academic exercise. Practical applications must be preceded by more academic evaluations of its effective validity, the user’s familiarity with de MCDA’s methodology and a cost-benefit analyses because the proposed evaluation model is time consuming and resource intensive. REFERENCES Anderson, P. H.; Lawton, L. (1990). “The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise.” Developments in Business Simulation and Experimental Learning, Vol. 17, 6-10. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Anderson, P. H.; Lawton, L. (1997). “Designing Instruments for Assigning the Effectiveness of Simulations”. Developments in Business Simulation and Experimental Learning, Vol. 24, 300-301. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Bana e Costa, C.A., Stewart, T.J., Vansnick, J.C. (1995). “Multicriteria decision analysis: some troughs based on the tutorial and discussion sessions of the ESIGMA meetings”. Euro XIV Conference, pp. 261-272, Jerusalem, July 3-6. Bana e Costa, C.A., Vansnick, J.C. (1997). “Applications of the MACBETH Approach in the Framework of an Additive Aggregation Model”. Journal of Multi-criteria Decision Analysis. Vol. 6, 2, 107-114. Beinat, E. (1995). Multiattribute Value Functions For Environmental Management. Amsterdam: Timbergen Institute Research Series. Bodily, S. E. (1985). Modern Decision Making: A Guide to Modeling with Decision Support Systems. New York: McGraw-Hill. Eden, C. (1988). “Cognitive Mapping”. European Journal of Operational Research . N. 36, p. 1-13. Ensslin, S. R. (2002). “A incorporação da perspectiva sistêmico-sinergética na metodologia MCDA- construtivista: uma ilustração de implementação”. Tese de doutorado – Programa de Pós-Graduação em Engenharia de Produção. Universidade Federal de Santa Catarina, Florianópolis. Ensslin, L; Montibeller, G. N.; Noronha, S. M. D. (2001). Apoio à Decisão: metodologias para estruturação de problemas e avaliação multicritério de alternativas. Florianópolis: Insular. Faria, A. J. (2001). “The changing nature of business simulation / gaming research: A brief history”. Simulation & Gaming ,Vol. 32, 1, 97-110. Feinstein, A. H.; Cannon, H. M. (2002) “Constructs of Simulation Evaluation.” Simulations & Gaming, Vol. 33, 4, 425-440. Gentry, J. W.; Dickinson, J. R.; Burns, A. C.; McGinnis, L.; Park, J. (2007). “The Role of Learning versus Performance Orientations when Reacting to Negative Outcomes in Simulation Games: Further Insights.” Developments in Business Simulation and Experimental Learning, Vol. 34, 4-10. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Gosenpud. J. (1990). “Evaluation of experiential learning. In James W. Gentry (Ed.). Guide to Business Simulation and Experimental Learning. (pp. 301-329). East Brunswick, NJ: Nichols/GP. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Goodwin, P.; Wright, G. (1991). Decision analysis for Management Judgement. Chichester: John Wiley & Sons. Keeney, R. L. (1992). Value-Focused Thinking: A Path to Creative Decision making. Cambridge: Harvard Univ. Press. Keeney, R. L., Raiffa, H. (1993). Decisions with Multiple Objectives - Preferences and Values Tradeoffs. Cambridge: Harvard Univ. Press. 569 pages. Keys, B., Wolfe, J. (1990). “The Role of Management Games and Simulations in Education and Research”. Journal of Management, Vol. 16, 2, 307-336. Landry, M. A. (1995). “Note on the Concept of ‘Problem’”. Organization Studies, Vol. 16, 2, 315-343 – EGOS 0170-8406/95 – 0016-0012. Larichev, O. I.; Moshkovich, H. M. (1997). Verbal Decision Analysis for Unstructured Problems. Amsterdam: Kluwer Academic Publishers. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 263 Roy, B. (1990). “Decision-aid and decision making”. in: Bana e Costa (ed.) Readings in Multiple Criteria Decision Aid, Berlin: Springer, 17-35. Schumann, P. L., Anderson, P. H., Scott, T. W., Lawton, L. (2001). “Framework for Evaluating Simulations as Educational Tools.” Developments in Business Simulation and Experimental Learning, Vol. 28, 215- 220. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Teach, R. (1990). “Profits: the false prophet in business gaming”. Simulation & Gaming, Vol. 21, 12-25. Teach, R. (2007). “Assessing Learning in a Business Simulation.” Developments in Business Simulation and Experimental Learning, Vol. 34, 76-8. Reprinted in the Bernie Keys Library, 8th edition [Available from http://ABSEL.org] Von Winterfeldt, D.; Edwards, W. (1986). Decision Analysis and Behavioral Research. Cambridge: Cambridge University Press. Washbush, J., Gosen, J. (2001). “An Exploration of Game- derived Learning in Total Enterprise Simulations”. Simulation & Gaming ,Vol. 32, 3, 281-296. Watson, S. R., Buede, D. M. (1987). Decision Synthesis. Cambridge: Cambridge Univ. Press. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 264 Table of Contents Volume 35, 2008 Linking Team Covenants To Peer Assessment Of Simulation And Experiential Performance Using the Balance Scorecard Approach: A Group Exercise IndoAmerican Enterprises Class Size and Game Design Implementation Of Effective Experiential Learning Environments Simulation Sensemaking: The BusinessWeek Approach To Effective Debriefing A Lesson in Hide-And-Go-Seek: A Team Building Game Thoughts On How To Motivate Students Experientially Experiential Learning Is Not Just Experiential Teaching: Measurement of Student Skill Acquisition via Assessment Centers ABSEL Redux: Reflections after a 25 Year Hiatus College Student's Expectations of Technology- Enhanced Classrooms: Comparing 1996 and 2006 The Business Student Satisfaction Inventory (BSSI): Development and Validation of a Global Measure of Student Satisfaction The "Big Picture Question" Project: Explorations in Teaching Creativity within a Force-Field Research Framework "Viva Voce": Oral Exams as a Teaching & Learning Experience Internships And Occupational Socialization: What Are Students Learning? Does Learning Occur in One-Shot, Non-Cooperative Games? Beliefs and Behavior, an Ancient Perspective and Modern Application Developing Enterprise Culture Among the Students Through Intercollegiate Competitions: A Case of Student Enterprise Competition (SEC) 2007 Student Views of Management Skills and Their Future Careers after Using Business Simulations Back to the Future: Gender Differences In Self-Ratings of Team Performance Criteria A Case for Experiential Learning: Using Central Europe as a Classroom Modeling Strategic Opportunities in Product-Mix Strategy: A Customer- Versus Product-Oriented Perspective Assessment in the Modern Large High-Tech Classroom Target Profit Pricing With the Web-Based Breakeven Analysis Package Are the Business Simulations We Play Too Complex? Shared Experience as Incentive for Horizontal Integration in Business Simulations Affinity Propagation: A Clustering Algorithm for Computer-Assisted Business Simulations and Experiential Exercises Marketing Simulation Results as Embedded Forms of Program Assessment Human and Agent Playing the "Beer Game" Issues in Porting a LAN-based Total Enterprise Simulation Game to a Web-based Environment Partners or Competitors? A B2B Simulation Evaluation Model of the Global Performance of a Management Simulation for the Academic Environment Applying Bloom's Revised Taxonomy In Business Games Do Price Strategies Work in Business Simulations? Early Japanese Gaming Simulation Efforts Using Par Players to Enhance Learning in Business Simulations Corporate Cartooning: The Art of Computerized Business Simulation Design Should Business Game Players Choose Their Teammates: A Study with Pedagogical Implications Goal Orientation and Simulation Performance Design and Demonstration of an Online Managerial Economics Game with Automated Coaching For Learning and Graded Exercises for Assessment Are Good Strategy Decisions Consistently Good? A Real-Time Investigation Active Learning 2.0 or Wiki is not a 4-letter Word Comparing Student Learning in Online and Classroom Formats of the Same Course How Do We Get To Tomorrow? The Path to Online Learning Similar Media Attributes Lead to Similar Learning Outcomes Facilitating Business Gaming Simulation Modeling