ASSESSING INDIVIDUAL PERFORMANCE IN A TOTAL ENTERPRISE SIMULATION Developments in Business Simulation and Experiential Learning, Volume 31, 2004 ASSESSING INDIVIDUAL PERFORMANCE IN A TOTAL ENTERPRISE SIMULATION Ricardo R. S. Bernard Universidade Federal de Santa Catarina – Brazil bernard@cse.ufsc.br ABSTRACT Although empirical studies have demonstrated there is no positive correlation between simulation performance and learning, in almost all business simulation exercises such indicator is used to evaluate participants. Considering that in real-world individual are usually evaluated by performance, and rarely by learning, it is justifiable to continue using simulation performance as an assessment indicator. Therefore, this paper extends existing literature presenting a methodology to assess individual performance in a total enterprise simulation course. Findings of first applications using such methodology are reported. Advantages of the devised methodology are also discussed. KEYWORDS: business simulation; assessment, individual performance. INTRODUCTION Assessment in business simulation courses is somehow controversial. At one hand, empirical evidence shows that the great majority of instructors using total enterprise simulations grade their participants based on simulation performance (Anderson and Lawtson, 1992b). On the other hand, authors argumentations (Anderson and Lawton, 1997; Teach, 1990; Thorngate and Carroll, 1987) and empirical evidences (Anderson and Lawton, 1992a; Wasbush and Gosen, 2001) have demonstrated that there is no relationship between simulation performance and learning. This paradox can be explained by Washbush and Gosen’s (2001: 292): … in real-world organizations, managers and employees are continually evaluated on performance and rarely on learning. In the university, we usually grade on mastery or performance via test or paper after the completion of a unit rather a change from one level of understanding, knowledge, or analytical ability to another. Grading on performance is what we usually do… Considering simulation performance will continue to be used in total enterprise simulations, a question arises: What are the most appropriated performance indicators? Profits, and other financial and economic indicators, are commonly used to evaluate simulation performance. However, as in real-world (Dearden, 1969; Eccles, 1991; Fisher, 1992; Kaplan, 1983; Ridgway, 1956), performance based only on such indicators has also been criticized in business simulations (Teach, 1990; Teach, 1997). In real-world many integrated and balanced measurements systems have emerged (Atkinson et al., 1997; Eccles and Pyburn; 1992; Kaplan, and Norton, 1992; Vitale et al., 1994). In simulated world some integrated performance systems have also been devised (Frizsche and Cotter, 1997; Thorelli, 1997). However, simulated performance systems continue to be used to evaluate simulated company as a whole, not individual performance. One exception is Teach’s (1997) work that has assigned individual indicators to each managerial function in a given business simulation. Four functions were considered in Teach’s model: manufacturing, marketing, comptroller, and executive. Comparisons are made using two indicators by function. Values from these two indicators are plotted on a graph along with the data for the other firms and distributed to all the firm’s specific functions. In doing so, each function can be evaluated against the industry standard or based upon their ranking with their counterparts in the other firms. Present paper also focuses on individual performance. However, it differs from Teach’s work in three aspects. First, Teach devised a performance measurement to be used in a specific business simulation. The methodology presented in this paper aims to be as generic as possible. Second, the business simulation used by Teach was developed to have no face-to-face contacts among team members during the decision making process. By contrast, this paper has specific dynamics to be applied in face-to-face meetings. Finally, in Teach’s work there is no emphasis on grading scores, as present paper does. This paper addresses individual performance to enrich literature in two aspects. First, in real-world integrated company performance systems are closely related to individual performance evaluations (Eccles and Pyburn; 1992; Ittner and Larcker, 1998; Otley, 1999). However, this is not the case of business simulation environment (Frizsche and Cotter, 1997; Thorelli, 1997). Second, using business simulations in regular basis since early nineties, this author has observed that, although teams are formed by individual functions, performances are usually evaluated by company performance. Consequently, individual efforts spent to achieve this performance are not considered. For example, a student has no effectively participated of the decision making process will receive the same score of the remaining team members. Therefore, introducing a grade based on individual performance can solve this problem. Additionally, it can also bring more realism to the business simulations because, as in real-world, trade-offs between functions is expected to be more accentuated during the decision making process. 197 mailto:bernard@cse.ufsc.br Developments in Business Simulation and Experiential Learning, Volume 31, 2004 METHODOLOGY TO ASSESS INDIVIDUAL PERFORMANCE The methodology to evaluate individual performance in business simulation courses assumes that two basic conditions exit. First, the decision making process is performed in teams, not individually. Second, business simulation is expected to provide indicators of performance to each function. At this regard, this methodology is more appropriated to be used in total enterprise simulations, also called top management games. Keys and Biggs (1990:49) define such simulations as: A total enterprise game is one which includes decisions in most of the main functions of business: marketing, production, finance, and personnel. Such games require integration of the various functional areas. In addition, total enterprise games incorporate environmental factors, such as general economic conditions and interest rates as important components of the learning experience. Once a team approach is preferred, and a total enterprise simulation is available, the methodology can be applied. Initially each participant is assigned to one team and to one managerial function such as finance, marketing, production, or personnel. Usually, these assignments can be random, self-defined, constrained self-selected, or defined by the instructor (Bacon et al., 2001). However, the methodology will privilege a self- selected assignment to managerial functions followed by a random assignment to team. When participants select their functions, confidence in the decision making process is expected to be higher. On the other hand, random assigning to teams is expected to create more heterogeneous groups. Associated with individual evaluations, this heterogeneity will proportionate a more real-world proximity because conflicts are expected to emerge more frequently in the decision making process. A one-to-one assignment of participant to function is preferred. Number of available team members superior or inferior to functions is not advised because it can influence individual assessments. The number of enterprises to be simulated is a logical strategy to solve this problem. A complementary strategy is creating the Chief Executive Officer – CEO function. Participants in this position will be responsible for coordinating the decision making process, and intermediating eventual conflicts between team members. Next step is defining performance indicators to each function. Considering practical aspects, all indicators should be extracted from reports issued by the business simulator. The number of indicators to each function will depend on the simulation complexity. However, three or four indicators are more advised. One or two indicators will make the assessment very sensitive. More than four indicators can turn the assessment very complicated to be managed with marginal gains. Table 1 shows a list of suggested indicators to each managerial function. Such indicators can be extracted from the majority of total enterprise simulations. As it can be seen, the CEO position, when existent, will continue to be assessed by company performance indicators. TABLE 1 – Performance indicators associated to functions Function Performance Indicator Assessment Market share (%) HB Sales growth (%) HB Sales ($) HB Marketing Demand to sales ratio (%) NZB Unit product cost ($) LB Productivity (number) HB Production programming NZB Production Employee motivation (scaling) HB Cash flow balance ($) LB Abnormal interest paid ($) LB Current liquidity ratio (%) HB Finance Debt to asset ratio (%) LB Employee turnover (%) LB Employee productivity (number) HB Motivation (scaling) HB Personnel Employee balance (necessary / existent) NZB Share value ($) HB Return on equity (%) HB Net profit margin (%) HB CEO Cumulative dividends ($) HB NOTE: HB = Higher Better; LB = Lower Better; NZB = Near Zero Better (negative and positive values are possible) 198 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Before scoring individual performance, weights have to be attributed to each indicator. Two strategies are indicated. A simpler strategy is weighting equally all indicators within a function. Another strategy is differentiating weights among indicators. The total sum of weights, however, must be 1.0, i.e. 100%, to each function. The weights can be altered by the instructor, tailoring them to meet specific needs, as also suggested by Frizsche and Cotter’s (1997) assessment tool. Once assigned participants to functions, defined indicators and attributed weights to each indicator, the next step is scoring individual performance. At least 3 options are possible. First option is scoring each indicator using a scaling raging from 1.0 to 10.0. Best performance in a given indicator receives score 10.0, while worst performance receive score 1.0. Remaining performances are assigned proportionate scores. For example, in a simulation with 3 companies attributed scores are 10.0 (best), 5.5, and 1.0 (worst). Second option is based on a discrete scaling, ranging from 1 to number of simulated companies (X). Worst performance is attributed score 1, while best performance receive the score related to X. Other performances are scored between 2 and X-1. In the two previous scales a constant gap exists between scores. Therefore, participants can estimate how many positions they can gain or losing in the next scoring, independently of the gaps among performances. However, as the gap between the best and the worst scores remains constant, distortions can arise; that is, lower performances will attributed the same score, no matter how distant they are from the top one. Third option avoids such distortions, assigning to top performance a score of 1.00, and remaining performances scores which represent the proportion to the top. This option is similar to Frizsche and Cotter’s (1997) suggestion to weight performance indicators. However, independently of the chosen option to score individual performance, individual score will be composed by the sum of scores achieved in each indicator. Table 2 presents a scoring example to the market share indicator in a simulation with 5 companies, considering the three cited strategies to assign scores. TABLE 2 – Scoring a specific indicator Company 1 2 3 4 5 Market share (%) 11 24 26 22 17 Ranking 5th 2nd 1st 3rd 4th Scoring – option A 1.00 7.75 10 5.50 3.25 Scoring – option B 1 4 5 3 2 Scoring – option C 0.42 0.92 1.00 0.85 0.65 NOTE: Option A = scaling 1 to 10; Option B = scaling 1 to X; Option C = ratio scale If an individual is absent in a given decision making process (considering the process is performed in classroom), it is possible to assign zero score to all indicators in this round to the absent individual. If this strategy is adopted, participants are expected to be more present in classrooms, because, otherwise, they will receive zero grades in each absent round. CEO, if existent, is advised to assume the function of the absent student, because a weak performance in a given function can prejudice company performance. Once indicators are scored, they are disclosed by round (e.g., by quarter) and cumulative, because integrating performance assessment using more than one indicator and over a series of quarters reduce the luck factor and the good- day, bad-day syndrome (Frizsche and Cotter, 1997). Tables 3 and 4 present examples of scoring by quarter and cumulative, respectively. TABLE 3 – Scoring an individual function by quarter Function Indicator Performance Ranking Score Market share (%) 22 2nd 4 Sales growth (%) 1 4th 2 Sales ($) 2.987.451 3rd 3 Marketing Demand to sales ratio (%) 13 1st 5 Total Score Company 2 14 NOTE: Scores in quarter 2, considering 5 companies and option B scoring. 199 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 TABLE 4 – Scoring individual performance Company Marketing Director Quarter 1 (score) Quarter 2 (score) Accumulated Score Ranking 1 Member A 9 7 16 5th 2 Member B 13 14 27 3rd 3 Member C 13 16 29 2nd 4 Member D 15 15 30 1st 5 Member E 10 8 18 4thº NOTE: Scores from quarter cells are extracted from Table 3, in row ‘Total Score’. Reassignment, the last methodology step, can be necessary for many reasons. For example, random assignments to teams can create a “bad” group (Bacon et al., 2001); or, considering educational purposes, it can be important to a participant perform more than one managerial function. Therefore, methodology has to be flexible to cope with these eventual adjustments. Because participants have individual scores, these adjustments can be easily managed. The only concern is that scores received by the participant in previous rounds must be always associated to him, no matter in which each new team, or function, he will be assigned. The methodology was devised to be as generic as possible. Eventually, some minor adjustments have to be done in performance indicators or scoring procedures. The methodology can be implemented using spreadsheets, or integrated to the simulation software. Latter option is preferable in terms of time savings. A step-to-step guide to the methodology is presented in Table 5. TABLE 5 – Methodology to assess individual performance in business simulation courses Step Activity 1 Assign participants to functions and teams 2 Define individual indicators of performance to each function 3 Weight indicators 4 Score individual performance 5 Show results of individual performance 6 Adjust team or function assignments PILOT PROJECT USING THE METHODOLOGY This methodology was tested in a pilot project involving 63 undergraduate students enrolled in required business simulation courses at Universidade Federal de Santa Catarina – Brazil, during the first semester of 2003. Students were originated from 3 groups. In one group a manufacturing simulation – SIND (2003) was used; while the two other groups used a retailing simulation – SIMCO (2003). Both simulations are top management games with more than 30 decision inputs per round. They are considered complex simulations according to the Keys and Wolfe’s (1990) definition. Team members were formed using self-selected assignment to functions and random assignment to teams. No adjustments were done during the courses in terms of changes in team or function assignments. Each team was composed by four members performing the following functions: CEO, marketing, finance, and personnel (in retailing simulations) or production/personnel (in manufacturing simulation). Business simulation courses were graded using managerial (50%) and academic performances (50%). Managerial performance is related to simulated business performance. Indicators to this performance were based on company performance (12.5% to share value and 12.5% to ROE); and individual performance (25% to indicators related to managerial functions). Grade system was adjusted to consider that all participants receive managerial performance grades sufficient to be succeed in the course; that is, the worst performances in share value, ROE, and functions will be always attributed the grade required to be approved in the course. In doing so, it will be assured that weak business performance will not be responsible to the student’s fails in the course. This strategy is consistent with the rationale that simulated business performance is not related to learning. Therefore, if a student failure occurs, it will be associated with academic performance. The indicators used to evaluate academic performance were oral debriefing (10%), written debriefing (15%), and participation in decision making sessions (25%). Courses were conducted during 9 simulated quarters, preceded by a practice round. Initial four quarters were simulated using traditional simulation performance evaluation; that is, return on equity and share value (an aggregated indicator composed by financial, economic, and market parameters). Individual performance was used in last five quarters. Indicators shown in Table 1 were used to assess individual performance. All indicators were equally weighted. Scores were attributed using option B strategy (worst performance = 1; best 200 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 performance = number of companies). Absent students in a given round were assigned the lowest scores in all indicators, independently of achieved performance in the quarter. Preliminary administrations of individual performance assessment in business simulation courses have shown stimulating results. A blind questionnaire was administrated after business simulation courses have finished to gathering participant’s perceptions about the grade system. Ninety-four percent of participants have answered to the questionnaire. In terms of member assignment to function and to team, 39 % considered the option chosen as ideal, 49 % preferred also selecting the team, 9 % preferred also random assignment to functions, and 3 % preferred to select the team and the instructor assign the functions. Figure 1 shows a graphic representation of ideal assignments of member to function and member to team from a team member viewpoint. A question was formulated to evaluate participant perceptions on how individual performance can be considered a good indicator of learning. Answers showed that 13% of respondents considered this indicator graded below that they expected, 84 % considered individual performance as a good indicator to grade learning; and 3% of participants reported individual performance indicator attributed more grade than they expected to receive. Grades below and above expectations can indicate that luck factor is also present in such indicator. Figure 2 shows a graphic representation of team members’ expectations about grades they should receive in terms of individual performance. FIGURE 1 – Team members preferences to select teams and functions Assignments were ideal Members also select teams Random assignment to functions members select teams and instructor the functions NOTE: In the exercise the functions were self-selected and teams were random. FIGURE 2 – Members expectations about grades received based on individual performance Below As expected Above NOTE: In terms of percentage. A five-point Likert scale (1 = less important; 5 = more important) was used to evaluate individual performance in relation to other traditional grade indicators. Individual performance achieved the second highest evaluation with a score of 4.3, slightly below the presence on decision making process (4.5). Other indicators were written debriefing (3.3), oral debriefing (3.7), return on equity (3.5), and share value (3.6). Figure 3 shows a graphic representation of the importance of each grade indicator used in the business simulation course according to its participants. 201 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 FIGURE 3 – Importance of grade indicators from a team member perspective NOTE: Using a five-point Likert scale (1 = less important; 5 = more important). From an instructor perspective it was observed a more individual engagement in decision making process after the individual performance was introduced. Trade-offs between functions were more apparent and conflicts emerged. Absenteeism drooped to virtually zero after the introduction of individual performance assessment. CONCLUSIONS Present methodology was not devised to substitute existing grade indicators, rather to complement them. Academic grades, such as oral and written debriefing, or traditional performance indicators, such as net income, return on sales, or return on assets, must coexist. Furthermore, individual performance and company performance should be integrated. In doing so, members can be faced with real dilemmas of making decisions in self-interest or in company interest. Therefore, main methodology contribution is providing an individual performance assessment, both with academic and practical implications. First empirical results were promising. Students were very confident about this additional performance measurement; and the instructor has added an instrument to assess students individually by their managerial performance. However, more studies are necessary to gather evidences of internal and external validity of this methodology, both in terms of representational and educational validity, as suggested in Feinstein and Cannon (2002). Future researches in the field can take many directions as follows: • Evaluate the relationship between individual performance and learning achieved in the specific function being managed. • Verify if the introduction of individual performance assessment improves simulated company performance as a whole. • Study the impact of changes in participant function within the same team on participant learning process. • Study self-interest versus company interest priority in decision making using individual performance. • Study the impact of participant changes among simulated companies (with or without function changes) on team motivation and cohesion. REFERENCES Anderson, P. H. & Lawton, L. (1992a) “The relationship between financial performance and other measures of learning on a simulation exercise.” Simulations & Gaming, 23, 326-340. Anderson, P. H. & Lawton, L. (1992b) “A survey of methods used for evaluating student performance on business simulations.” Simulations & Gaming, 23, 326-340. Anderson, P. H. & Lawton, L. (1997) “Designing instruments for assigning the effectiveness of simulations.” Developments in Business Simulations and Experiential Learning, 24, 300-301. Atkinson, A. A., Waterhouse, J. H. & Wells R. B. (1997) “A Stakeholder Approach to Strategic Performance to Measurement.” Sloan Management Review, Spring, 25- 37. Bacon, D. R., Stewart K. A., & Anderson E. S. (2001) “Methods of assigning players to teams: A review and novel approach.” Simulations & Gaming, 32, 1, 6-17. Dearden, J. (1969) “The case against ROI control.” Harvard Business Review. Mai – June, 124-135. Eccles, R. G. (1991) “The Performance Measurement Manifesto.” Harvard Business Review, Jan-Feb., 131- 137. Eccles R. G. & Pyburn P. J. (1992) “Creating a Comprehensive System to Measure Performance.” Management Accounting. October, 41-44. Feinstein, A. H & Cannon, H. M. (2002) “Constructs of simulation evaluation.” Simulations & Gaming. 33, 4, 425-440. Fisher, J. (1992) “Use of Nonfinancial Performance Measures.” Journal of Cost Management. 6, Spring, 31-38. Fritzche, D. J. & Cotter, D. (1997) “Business policy game: An international simulation - An assessment tool.” Developments in Business Simulations and Experiential Learning. 24, 25-30. 0.0 1.0 2.0 3.0 4.0 5.0 Decision making Written debriefing ROE Oral debriefing Function Share value 202 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Ittner, C. D. & Larcker, D. F. (1998) “Innovations in performance measurement: Trends and research implications.” Journal of Management Accounting Research. 10, 205-238. Kaplan, R. S. (1983) “Measuring Manufacturing Performance: A New Challenge for Managerial Accounting Research.” The Accounting Review. 58, 4, 686-705. Kaplan, R. S. & Norton, D. P. (1992) “The Balanced Scorecard – Measures That Drive Performance.” Harvard Business Review. Jan – Feb.: 71-79. Keys, B. & Biggs, W. (1990) “A review of business games.” In J.W. Gentry (ed.). A guide to business gaming and experiential learning. East Brunswick, NJ: Nichols/GP. Keys, B. & Wolfe, J. (1990) “The role of management games and in simulation in education and research.” Journal of Management, 16, 2, 307-336. Otley, D. (1999) “Performance management: a framework for management control systems research.” Management Accounting Research. 10, 363-382. Ridway, V. F., (1956) “Dysfunctional consequences of performance measurement.” Administrative Science Quarterly, 240-247. SIND – Manufacturing Simulation. Bernard Sistemas (2003). Florianópolis, Brazil: www.bernard.com.br SIMCO – Retailing Simulation. Bernard Sistemas (2003). Florianópolis, Brazil: www.bernard.com.br Teach, R. (1990) “Profits: the false prophet in business gaming.” Simulations & Gaming, 21, 12-25. Teach, R. (1997) “The design of an Internet game.” Developments in Business Simulations and Experiential Learning, 24, 336-343. Thorelli, H. B. (1997) “Musings on business game performance evaluation.” Developments in Business Simulations and Experiential Learning, 24, 12-18. Thorngate, W. & Carroll, B. (1987) “Why the best person rarely wins.” Simulations & Gaming, 18, 299-320. Vitale, M., Mavrinac, S. C., & Hauser, M. (1994) “New Process/Financial Scorecard: A Strategic Performance Measurement System.” Planning Review, Jul-Aug, 12- 17. Washbush, J. & Gosen J. (2001) “An exploration of game- derived learning in total enterprise simulations.” Simulations & Gaming, 32, 3, 281-296. 203 Table of Contents Volume 31, 2004 Controlling the Complexity and Orenting Target Groups by a Modular, Server-Based Business Game System Learning Network Demonstration: Delivering Business Education in a Distance Learning Environment Economic Evolution, Human Capital Investment, and Adult Distributed Electronic Learning: A Literature Review Designing a Globalization Simulation to Teach Corporate Social Responsibility Developing and Teaching an Online / In-Class Hybrid: A Demonstration A Model for Evaluating Online Instruction An Evaluation of a Distributed Learning Course: A Students'-Eye Perspective Blended Learning Strategy Improved Business Writing Skills How to Receive and Process Attachemnts while Greatly Reducing the Risk of Viruses and Trojans Introducing Online Components to a Class: How to Increase teh Likelihood of Success Teaching Strategic Communications Online: Using Learning Outcomes to Develop a Case-Based Course Implementing Distance Approaches to Education: A Panel Discussion for ABSEL: Las Vegas, 2004 MANDI: Learning Management Through Field Sales Experience An International Capital budgeting Experiential Exercise A Primer To Combating Terrorism: Playing It Safe While On Overseas Assignment (An Experiential Exercise) Integrating The Business Curriculum With A Comprehensive Case Study: A Prototype The Case Brief: A Model For Case Analysis, Writing And Discussion Technology Infused Pedagogy And Delivery – A Sure Bet? A Proposal For Panel Discussion Absel Conference 2004 Research Strategy And The Bkl: Getting The Most From The Absel Archives Simple But Effective: Rediscovering The Class Discussion Needle And Thread: An Activity For Examining Various Management Behaviors A Customized Excel Data Analysis System For Use In Undergraduate Marketing Research Team Leader Selection - Does It Matter? The Power Of Perspective: Reframing Your Framing Skills For Innovative Instruction In Leadership And Influence Exercise: How Should Merit Raises Be Allocated? An Online Situation For Problem-Based Learning In A Junior-Level Management Course The Eden Alternative As A Roadway For Change: A Service Learning Quality Improvement Project Avoiding Catastrophe: The Role Of Individual Accountability In Team Effectiveness Omega Systems: A Change Management Exercise The Risks And Rewards Of Providing Students A Structured Cheating Opportunity Experimentation With Assessment Techniques: A Proposal For Panel Discussion Using A 2 - Page Case To Introduce Concepts Of Business Strategy Interactive Session The Integration Of Appreciative Inquiry And Experiential Learning For Peak Performance Appreciative Inquiry Case Story: New York City Leadership Challenge Individual Achievement Versus Team Performance: An Empirical Study With Business Games Some Strategists Don't Learn Or Can't Learn Computer Simulation: A Design Architectonic On The Value Of Bugs In Simulation Environments Online Sales Forecasting With The Multiple Regression Analysis Data Matrices Package Simulation Exercises And Problem Based Learning: Is There A Fit? A Study Of Business Game Stock Price Algorithms Assessing Individual Performance In A Total Enterprise Simulation Information Use In A Business Game Determining The Value Of A Firm Unsorting Algorithms For An Ordered List And Its Application To Business Simulations Teaching Public Finance Management Through Simulation Antecedents Of Game Performance Student Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Implementation And Impacts Of The Balanced Scorecard: An Experiment With Business Games Impact: Shocking The Legacy Mindset Implementation Of The Eepad Framework Of Business Processes In An Accounting Information Systems Course Are Business Games Really Delivering What Students Are Led To Believe?? Reporting Lessons Learned: What Gets Reported; Who Gains Value Teacher Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Student Reactions To The Use Of A Computer-Based Simulation As An Integrating Mechanism For A Mba Curriculum A Cognitive Investigation Of The Internal Validity Of A Management Strategy Simulation Game The Casino Challenge: Making Simulation Delivery A Safe Bet! Accounting For Company Reputation: Variations On The Gold Standard Foreign Currency Hedging: A Simulation The Influence Of Variables Easily Controlled By The Instructor/Administrator On Simulation Outcomes: In Particular, The Variable, Reflection. Absel Awareness Among Business School Faculty Validating Business Simulations: Does High Market Share Lead To High Profitability? Simulation Debriefing Procedures Coaching And Business Simulations: A Formula For Success? A Seminal Inventory Of Basic Research Using Business Simulation Games The Influence Of Scorecard Evaluation On Decisions And Outcomes