IMPLEMENTATION AND IMPACTS OF THE BALANCED SCORECARD: AN EXPERIMENT WITH BUSINESS GAMES Developments in Business Simulation and Experiential Learning, Volume 31, 2004 IMPLEMENTATION AND IMPACTS OF THE BALANCED SCORECARD: AN EXPERIMENT WITH BUSINESS GAMES David Kallás University of São Paulo david@symnetics.com.br Antonio Carlos Aidar Sauaia University of São Paulo asauaia@usp.br ABSTRACT Treated initially as a performance measure system, the balanced scorecard has evolved and today it is possible to state that it is a management tool. It presents an ordination of previous concepts and ideas in logic, objective and intelligent form. A study was performed with 32 simulated companies. In the experiment, the balanced scorecard was implemented in 5 companies and the results were compared to the remaining 27. It was observed with statistic significance that the experimental group (the 5 simulated companies) performed better than the control group (the other 32) in terms of the game total score. KEYWORDS: strategic management, strategy, strategic planning, business games INTRODUCTION From Taylor’s Scientific Management School until nowadays, concepts have been evolved in a vertiginous rhythm. Constant launching of theories had led to a diversity of new proposals of business management tools (FERREIRA et al., 2002:3). According to ROSENBURG (2001), many of the 10,000 worldwide-published business books from 1998 to 2001 treat new ways to manage business. Their validity is incontestable, as a rule, though being a responsibility of the most aware manager to notice in which degree these contributions differ, under the several given names, and whether these new proposals might or should be adopted in his/her company. A typical case of analysis is the balanced scorecard. Originally presented by KAPLAN & NORTON (1992) as a tool to monitor corporate performance, its potential has developed and turned out to be seen as a strategic management tool for large, medium and small-size organizations, also including the possibility for individual and team application. Soon after its creation, the concept was widely disseminated and implemented by executives all over the world. The efficiency of the method has elicited distinct opinions among the companies due to the different context in which each application was implemented. These apprehensions, grown in both the academic and professional environment, have motivated this study to be performed. Additionally to this fact, there is an inherent wish that this study may help organizations to become more and more competitive in the international and domestic scenario, broadening their knowledge of management tools. REVIEW OF LITERATURE The balanced scorecard (BSC) may not be derived from strategic management concepts. Its origin is related to the limitation of the traditional performance measurement methods (KAPLAN & NORTON, 1992:71), what in fact is one of the strategic planning problems pointed out by ANSOFF et al. (1976:8). Nevertheless, following its evolution, the instrument has become an important strategic management tool. More than a trivial measurement exercise, the BSC motivates breakthrough improvements in critical business areas, such as product development, internal processes, customers and marketing (KAPLAN & NORTON, 1993:134). The balanced scorecard is a management tool that materializes the corporate vision and strategy through a coherent map that includes goals and performance measures organized according to four distinct perspectives: financial perspective, customer perspective, internal processes perspective, and learning & growth perspective. These measures should be interconnected in order to communicate a small number of general strategic issues, such as the corporate growth, risk reduction or productivity enhancement (KAPLAN & NORTON, 1997:24-25; 44). After the onset of this management tool and initial applications to North American corporations, not only authors but also executives realized that its scope had expanded the original concepts (JÚLIO & NETO, 2002:181; CAMPOS, J., 1998:64). Concurring with this view, KAPLAN & NORTON (2000a:18) noticed that successful corporations that applied the balanced scorecard solution revealed a consistent pattern in the provision of focus and strategic alignment. “Although each organization had approached the challenge on its own way, with different rhythms and sequences, we observed the occurrence of five common principles, which we called principles of strategy- focused organization”: • Principle #1: Translate strategy into operational terms: it is not possible to implement a strategy without describing it first. Strategy mapping and the balanced scorecards take care of deficiencies in tangible assets measurement systems of the industrial era. The links in the measurement of cause 242 mailto:david@symnetics.com.br mailto:asauaia@usp.br Developments in Business Simulation and Experiential Learning, Volume 31, 2004 and effect relations at the strategy mappings show how intangible assets turn into (financial) tangible results. The scorecard schema uses quantitative, but non-financial indicators (such as cycle times, market share, innovation, satisfaction and competence), what makes possible the description and measurement of the value creating process, instead of a trivial inference (KAPLAN & NORTON, 2000a:20-21). • Principle #2: Align the organization to create synergies: this principle represents the corporate BSC, cascaded for the business and shared services units of the organization according to the vision of internal processes. • Principle #3: Transform strategy in everyone’s job: strategy-focused organizations require that all employees understand the strategy and conduct their daily routine jobs in order to contribute to accomplish its goals. In many cases, individual scorecards were also implemented to define personal goals. Finally, each one of the successful organizations linked the incentives program and rewarding system with the balanced scorecard (KAPLAN & NORTON, 2000a:22-23). • Principle #4: Make strategy a continuous process: successful corporations in adopting the balanced scorecard implemented a strategic management process. The so-called double-loop process integrates tactical management (financial budgets and monthly evaluation reports) with strategic management into a single continuous and unceasing process. • Principle #5: Executive leadership to mobilize change: A successful balanced scorecard solution begins by realizing that it is not a measurement project but a changing program. In the initial stage, the focus is positioned to mobilize and generate impulse to launch the process. After mobilizing the organization, the focus is re-calibrated on governance (EPSTEIN & WISNER, 2001). And finally, a new management system is gradually developed – a strategic management system that institutionalizes the new cultural values and new structures in the new management model. The new subsequent stages may be carried out in a two or three years’ period (KAPLAN & NORTON, 2000a:26-27). An expressive contribution of balanced scorecard is the alignment of lag indicators with lead indicators in a logic form and aligned to the strategy. STERN STEWART (1999) recognizes that the BSC can be complemented with other strategic and tactic management tools, such as the Activity- Based Costing (ABC) and Economic Value Added (EVA). The ABC helps managers to better understand the costs and capital structure, while BSC schema enlarges the management view of non-financial matters and EVA drives managers to create value. This view is also shared by SHINOHARA (2003:11). It can be stated that in the analysis and implementation of the strategy, the BSC considers different stakeholders. The stakeholders analysis provides elements to compare several perspectives and come to a decision, which is a tool also used by SAUAIA & KALLÁS (2003) to analyze the “cooperate for profits or compete in oligopolistic markets” dilemma. Another benefit is related to the communication of the strategy within the organization (YOUNG & O’BYRNE 2001:291). The BSC describes the corporate vision of the future for all the organization in order to establish shared goals. A holistic model of strategy is created, showing to all employees how they can contribute to achieve organizational success (KAPLAN & NORTON 1997:154). CAMPOS, J. (1998:105) argues that another BSC’s benefit is related to focus on business actions. Although the balanced scorecard provides senior executives with additional measures, it minimizes the amount of information to be analyzed by focusing on the most critical matters and restricting the number of indicators to be used. Despite its limitations, strategic planning is by far the most popular tool used within the organizations, a particularly powerful one if allied to the balanced scorecard. Annual studies performed by BAIN & COMPANY (2001) search to verify which are the most popular management tools in Brazil and in the world, as well as their satisfaction level. According to the Brazilian results, strategic planning is the most applied management tool and the second by satisfaction level. On the other hand, the balanced scorecard is a recent tool that is rapidly spreading out. Its application increased to 56% of total responses in 2001 as compared to 30% registered in 2000. Considering the satisfaction aspect, it ranked fourth with a 4.29 grade in a maximum of 5.0. Some criticisms have been voiced in literature. YOUNG & O’BYRNE (2001:301) observe that some BSC users tend to confuse the means with the ends. Investment in customers, supplier and employee relationship management are not corporate goals, but means to aggregate value to shareholders. When managers forget this fundamental aspect, the balanced scorecard may become an excuse to defend the organization’s failure to perform higher financial results. BOYETT & BOYETT (1999:269) alert about the connection matter between goals and BSC indicators: “In real world, the association between cause and effect is rarely so clear. In most situations we should be satisfied by only including a good many of the right measures in the scorecard, without trying to figure out the relation among them”. KAPLAN & NORTON (1996, 1997, 2000a, 2000b) recognized such limitation several times, when they stated that the BSC built by a company is an initial hypothesis. The business strategy defines a rationale of how value will be created to the shareholders. It defines actions and resources required to meet the targeted results. As such, it is based on a group of premises that must be tested (NORTON, 2001:1). An excellent set of measures does not guarantee itself a winning strategy. The failure in converting operational performance into financial results must send executives back to the “drawing table” to rethink their business strategy or their strategy implementation plan (KAPLAN & NORTON, 1992:77). There are many attempts to complement the simplicity of BSC with more complex models. AKKERMANS & OORSCHOT (2002) suggest complementing BSC with system dynamics methodology, just to mention one of them. RESEARCH PROBLEM Based on the wide acceptance of BSC and the concerns of several authors regarding its actual contribution, the burning question that arises is the following: Does the balanced 243 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 scorecard application have an impact on corporate success indicators? Success will be measured through an experiment and a comparative analysis of the seven corporate performance indicators within the experimental environment of the MMG – Multinational Management Game, (KEYS et al., 1992) shown in Exhibit 1. The purpose is to reject the following hypothesis H0 (the experimental application of the balanced scorecard in simulated companies does not produce positive impact on their success indicators). Exhibit 1 - Success Indicators of the experiment Indicator Description Return on Equity (ROE) Net profit of the period divided by average equity. Return on Sales (ROA) Net profit of the period divided by net sales. Return on Assets (ROA) Net income divided by average assets. Market Share Weighted average of the market share of products. Asset Turnover Net income of the period divided by average assets. Inventory Turnover Costs of products sold in the period divided by the average between initial and final stock. Debt to Total Assets Total debt over total assets. Total Score Overall score based on the sum of relative rank performance of each of the seven indicators above. RESEARCH METHOD Given the validity and invalidity points of each type of experiment, and all environmental features of the business games considered, the chosen experiment was: “to pre-test and post-test with casual experimental and control groups”. According to CAMPBELL & STANLEY (1979:16), this is one of the most recommended methods. LAKATOS & MARCONI (2000:242-243) state that this method employs two paired groups. That one, in which an experimental variable is introduced, is denominated experimental group, while the other group that is under no influences works as the control group. Both groups are observed simultaneously, but no experimental variable is introduced into the control group. For this reason, the difference between measures taken after and before in the control group (O’2 - O’1) is supposed to be the result from actions of uncontrollable variables or factors, while, in the experimental group (O2 - O1), the same difference corresponds to the action of the experimental variable plus the same uncontrollable events observed in the control group. Thus, the experimental variable effect can be determined by subtracting the difference of the two measures in both the control group and the experimental group: [(O2 - O1) - (O’2 - O’1)]. DESCRIPTION OF THE EXPERIMENT Two groups were evaluated, an experimental and a control one, formed by students of the discipline “Business Games” (EAD-472) offered in the graduate program of Business Administration and Accounting courses at FEA/USP – Brazil. In this discipline a fictitious market simulation is run, where the companies, formed by students, define prices, investment volume, wages, and other identical decisions taken in real companies. These simulations are compiled in the MMG software that simulates market conditions from certain parameters and generates the results for the teams, which, based on these results, will make their new decisions to the next period. This research analyzes the results of two groups, one of them (experimental group) formed by teams that used the balanced scorecard. The analysis was performed through the comparison of results between the groups that used this method (X) and those that did not use it (O), all based on the success indicators described at Exhibit 1 above. The BSC application in the simulated companies of the experimental groups started between run 2 and 3, out of 7 runs of the total game application. The balanced scorecard application process was similar for all the companies. The companies spent the same amount to pay the consultant, though within different commercial conditions (resulting from an uneven negotiation effort among them). The BSC design process (strategy map building and selection of strategic indicators) was performed in a relatively short period of time (immediately after the acquisition of the consulting services). The rationale to build the BSCs was the same, as well as a limited number of goals and indicators was maintained in order to preserve the simplicity, as recommended in the method. No BSC had less than 10 or more than 14 strategic objectives (average of 11.4 objectives), as well as an average of 1.54 indicators was identified for each objective. This average is in compliance with the recommendation of KAPLAN & NORTON (2000a:393) that the BSC should have an average of 1.5 indicators per objective. 243 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Exhibit 2 illustrates one of the five BSCs of the experiment (a company called “Antarctic Tech”), in which the boxes represent the strategic objectives for each perspective, and beside each objective its strategic indicators is showed. For this case, there are 21 strategic indicators. From the seven performance indicators that compose the overall total score, only 5 appear in Antarctic Tech’s BSC, indicating its strategic focus. 243 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Exhibit 2 – Balanced Scorecard for the “Antarctic Tech” Company ROE ROA ROS Debit over total Income Profit by client Finantial perspective Price / Market average price for products A and B Quality Image Frequency of extra-class meetings Amount of time spent between classes (h) Learning and Growth Perspective Maintain debt at adequated level Enchance liquidity Mission: Enhance client productivity by supplying highly quality and performance equipments High performance team Have excelent R&D processes Enhance Profitability Acquire profitable clients Asset Turnover Productivity by ws A and B Client perspective Internal Processes Perspective Offer high value product to the client Improve client service level Produt. A Produt. B P&D / Income Advertising/Income Advertising Commision for sellers Inventory turnover turnover/abs Wages paid / market average Maintain low inventory level Employees satisfaction ANALYSIS OF RESULTS The quantitative analysis of BSC effects on the indicators of the simulated companies was carried out by comparing the performance deltas measured before and after the BSC implementation to the experimental group (EG) and control group (CG). In this case, the analysis of the difference between before and after was executed through repeated measures of related data, according to Wilcoxon and Friedman non- parametric tests. To analyze the experimental and control groups, a comparison of the independent averages was performed; in this case, the Mann-Whitney non-parametric test was applied (STEVENSON, 2001:307-334), once the sample size did not permit to apply the parametric t- test as suggested by CAMPBELL & STANLEY (1979:42). For the analysis purposes, a significance level of 5% was considered. Exhibit 3 presents a summary of the main results of the mentioned non-parametric tests. Exhibit 3 – Success Indicators of the experiment Analysis 7 paired variables CG 7 paired variables EG Delta after – before CG x EG before x after CG before x after EG Non-parametrical test Friedman Friedman Mann-Whitney Wilcoxon Wilcoxon Total Score 0.978 0.008 0.046 0.665 0.043 Inventory turnover 0.003 <0.001 0.113 0.001 0.043 Assets turnover 0.028 <0.001 0.068 0.011 0.043 Debt to Total Assets 0.702 0.070 0.046 0.131 0.043 Return on sales (ROS) 0.094 0.353 0.026 0.009 0.343 Return on equity (ROE) 0.885 0.154 0.241 0.428 0.138 Return on assets (ROA) 0.804 0.155 0.109 0.343 0.138 P of si gn ifi ca nc e Market share 0.930 0.244 0.263 0.683 0.174 When looking at each of the seven indicators individually, only two of them (debt to total assets and return on sales) are statistically significant (Mann-Whitney test). However, the aggregated total score, based on the relative performance of all the seven indicators, is also statistically significant. This shows that the BSC had impacted most the overall performance then individual performance indicators. The Exhibit 4 shows the evolution of one of the previous indicator, the total score for the multinational management game. The performance difference is clearly noticed. In the first two rounds the average score of Control Group (27 companies) showed higher than Experimental Group. Five companies started implementing BSC from the third year on. The average score of Experimental Group became steadily higher in the year four and increased consistently until the year 6. In the last year of the 245 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 experiment the score shows a slight reduction that could be explained by end-of-game strategies of teams. Exhibit 4 – Evolution of total score for the game (experimental x control group) 220 270 320 370 420 score year 1 score year 2 score year 3 score year 4 score year 5 score year 6 score year 7 average experimental group average control group Concerning the individual analysis of BSCs and correlations among indicators, some discussions are brought forward: • Each BSC represents a unique strategy: through the strategic mapping readings it was possible to clearly notice each one of the general strategies chosen by the simulated companies. This fact denotes that each one of the five strategies is distinct in relation to the others. This perception corroborates with PORTER (1996) when he states that the business strategy is defined by a unique corporate positioning, its choices and options coping with competition. • There is a common base to all companies in relation to determined strategic objectives and key performance indicators: according to the analysis, 3 indicators appear in all BSCs (R&D/Net Sales, productivity A and B), other 3 appear in at least 4 BSCs (turnover, price / market average, ROE) and another 7 appear in at least 3 BSCs. When dividing these 13 indicators by the average of the 17 indicators per BSC, it is verified that, on average, 76% of BSC indicators appear in at least 3 out of 5 simulated companies. This may suggest that: o There is a logic and common base in any and every business strategy, related to general aspects of corporate management, such as, increasing revenues, reducing costs, improving productivity, etc. This information meets the understandings of KAPLAN & NORTON (1996) that state the existence of a common base of approximately 80% of BSCs for similar business industries. o The singularity of each organization strategy remains in the emphasis of each goal to the same indicators and in the choice of the remaining 24%. o The experiment environment, due to the limited algorithms in the simulator, might have reduced the list of possibilities for choosing alternative indicators. • BSC application produced positive impact on the total score, but not on all success indicators: according to the statistics analysis performed, it is possible to state that the variation of score obtained before vs. after the experiment application was higher in the control group, considering a 5% significance level. Concerning the indicators that compose the score, this analysis is confirmed only for “debt to total assets” and “return on sales”. If the significance level were increased to 12%, the tests for “inventory turnover”, “assets turnover” and “return on assets” would be accepted. Taken these data into consideration, the following analysis are suggested: o The score is calculated in a comparative form, which can cause distortions in this indicator. As mentioned before, the score is distributed according to a comparative corporate performance ranking per indicator. Thus, to each indicator, a company could obtain from 10 to 80 points, according to its comparative position. This linear distribution can hide performances distributed in a non-linear way. o The strategies of each company, in spite of being distinct, had common aspects (reduction in indebtedness, elevated margins, high inventory turnover, and high assets turnover). The “market share” indicator, for example, despite being one of the seven simulator success indicators, appeared only in 2 out of 5 BSCs. Since strategy is a synonym of choice (KAPLAN & NORTON, 2000a:102), each company might have focused on distinct success indicators, but all of them aimed better overall results. 1 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 o Extreme data partially compromised the analysis of “inventory turnover” indicator. Some companies reduced their stocks to zero from year 4, maintaining this level on subsequent years. As the inventory turnover is calculated by dividing cost of goods sold by average stock, some indicators resulted in too high values that compromised the analysis to a certain point. In the light of the previous discussion, it is possible to reject H0 (the experimental application of the balanced scorecard in simulated companies does not produce positive impact on their success indicators). Other factors that not the BSC application could have happen, but also in the control group, due to the nature of the experiment. CONCLUSIONS The combination of business games with balanced scorecard is a fresh initiative that draws the attention of researchers of games and simulations. Firstly introduced by SAUAIA (2001), it was later discussed by DICKINSON (2003) and PRAY et al. (2003). The results of the research demonstrated that the experimental application of the balanced scorecard in simulated companies might have improved its success indicators. Simulated companies that adopted BSC seemed to perform better as a consequence of practiced BSC principles. This technique could be included in the business courses curriculum and offered as a relevant alternative tool for business management. Learn by doing can make students realize the positive consequences of using BSC, and make them aware of the importance of management instruments. Tools like BSC should be tested with different business simulation and should be practiced in academic environment before exposing managers to unnecessary risk in the real world. This seems to be one more opportunity for adopting business games in education and research. LIMITATIONS AND PROPOSITIONS According to CAMPBELL & STANLEY (1979:6), if the experiments are successful, they will need a reply and a cross- validation in different times, under different conditions, before they can be theoretically interpreted with confidence. Moreover, although experimentation is recognized as the basic proof language and as the only judgment that can clear out the doubts of rival theories, it must not be expected that “crucial experiments”, which challenge opposite theories, have to necessarily generate transparent results. For future studies, it is proposed to reply the experiment with the following recommendations: o Use other business management simulators to eliminate the calibration problem (CAMPBELL & STANLEY, 1979). o Enlarge the sample to perform parametrical statistic tests. o Diversify the participants’ profile, considering the possibility of applying the experiment to executive courses and MBAs, in which the managers of simulated companies are actual corporate managers. As yet, the balanced scorecard concept presents blanks and opportunities of evolution and development. Nevertheless, the corporate application in this experiment was demonstrated to be effective. We expect that through academic and professional works the positive and negative aspects can be managed and discussed in order to accomplish the maturity of the concept. BOYD & WESTFALL (1978:101-103) apud LAKATOS & MARCONI (2000:243) consider that, if the procedure “pre-test and post-test applied to casual experimental and control groups” is proven to be true for studies in which the subjects are inanimate beings, some limitations can occur when human capital is the object of research. They exemplify by indicating that the before measure, when performed with people, can lead them to pay more attention to the object of the study, thus bringing forward two distinct and opposite attitudes: to be more affected by the experimental variable or crystallize the oppositions. The educational effect may motivate people and drive them to observe the object of investigation more carefully, and could affect the control group as well. REFERENCES Akkermans, H.; Oorschot, K. (2002) Developing A Balanced Scorecard With System Dynamics. Available At: Http://Www.Minase.Nl/Pdf/Balanced.Pdf. Acessed On 01/11/2002. Ansoff, H. I.; Declerk, R. P.; Hayes (Org). (1976) From Strate- gic Planning To Strategic Management. 4. Ed. Vandelbilt University: John Wiley & Sons, 1976. Bain & Company. (2002) Vencedoras Confirmadas. Hsm Management, São Paulo, Ano 6, N. 31, P. 138-142, Março- Abril 2002. Boyett, J. H.; Boyett, J. T. (1999) O Guia Dos Gurus: Os Melhores Conceitos E Práticas De Negócios. Rio De Janeiro: Campus, 1999. Campbell, D. T.; Stanley, J. C. (1979) Delineamentos Experimentais E Quase-Experimentais De Pesquisa. São Paulo: Epu, 1979. Campos, J. A. (1998) Cenário Balanceado:Painel De Indicadores Para Gestão Estratégica Dos Negócios. São Paulo: Aquariana, 1998. Dickinson, J. R. (2003) “The Feasibility Of The Balanced Scorecard For Business Games.” Developments In Business Simulation And Experiential Learning, Baltimore, Volume Thirty, 2003, 90-98. Espstein, M. J.; Wisner, P. S. (2001) “Increasing Corporate Ac- countability: The External Disclosure Of Balanced Score- card Measures.” Balanced Scorecard Report. Harvard Busi- ness School Press, Volume Three, Number 4, August 2001, 10-3. Ferreira, A. A.; Reis, A. C. F.; Pereira, M. I. (2002) Gestão Empresarial: De Taylor Aos Nossos Dias: Evolução E Tendências Da Moderna Administração De Empresas. São Paulo: Pioneira Thompson Learning, 2002. Júlio, C. A.; Neto, J. S. (Org). (2002) Estratégia E Planejamento: Autores E Conceitos Imprescindíveis. São Paulo: Publifolha, 2002. Kaplan, R. S.; Norton, D. P. (1997) A Estratégia Em Ação: Balanced Scorecard. 4. Ed. Rio De Janeiro: Campus, 1997. 247 http://www.symnetics.com.br/artigos/default.asp. Acessado em 13/10/2002 http://www.symnetics.com.br/artigos/default.asp. Acessado em 13/10/2002 http://www.symnetics.com.br/artigos/default.asp. Acessado em 13/10/2002 Developments in Business Simulation and Experiential Learning, Volume 31, 2004 Kaplan, R. S.; Norton, D. P. (2000) “Having Trouble With Your Strategy? Then Map It.” Harvard Business Review. Boston, Volume Seventy-Eight, Number Five, September-October 2000b, 167-76. Kaplan, R. S.; Norton, D. P. (2000a) Organização Orientada Para A Estratégia: Como As Empresas Que Adotam O Balanced Scorecard Prosperam No Novo Ambiente De Negócios. Rio De Janeiro: Campus, 2000a. Kaplan, R. S.; Norton, D. P. (1993) Putting The Balanced Scorecard To Work. Harvard Business Review. Boston, V. 71, N. 5, P. 134-147, Setembro-Outubro 1993. Kaplan, R. S.; Norton, D. P. (1992) The Balanced Scorecard – Measures That Drive Performance. Harvard Business Re- view. Boston, V. 70, N. 1, P. 71-79, Janeiro-Fevereiro 1992. Kaplan, R. S.; Norton, D. P. (1996) Using The Balanced Score- card As A Strategic Management System. Harvard Business Review. Boston, V. 74, N. 1, P. 75-85, Janeiro-Fevereiro 1996. Keys, J. B.; Edge, A. G.; Wells, R. (1992) A. The Multinational Management Game – A Game Of Global Strategy. 3. Ed. Dallas: Business Publications Inc, 1992. Lakatos, E. M., Marconi, M. A. (2000) Metodologia Científica. 3. Ed. São Paulo: Atlas, 2000. Norton, D. P. (2001) Building Strategy Maps: Testing The Hy- pothesis. Balanced Scorecard Report. Harvard Business School Press, V. 3, N. 1, P. 1-4, Janeiro-Fevereiro 2001. Porter, M. (1996) “What Is Strategy.” Harvard Business Review. November-Dezember 1996. Pray, T. F.; Perotti, V.; Vaughan, M. J. (2003) Using The Bal- anced Scorecard To Improve Strategic Planning And Effec- tive Decision Making Within Simulations. Developments In Business Simulation And Experiential Learning, Baltimore, Volume Thirty, 2003. Rosenburg, C. (2001) Pense Antes De Sacar O Canivete: Para Que Servem As Ferramentas De Gestão. Revista Exame, Outubro De 2001. Sauaia, A. C. A. (2001) “Evaluation Of Performance In Business Games: Financial And Non Financial Approaches.” Devel- opments In Business Simulation And Experiential Learning, San Diego, Volume Twenty-Seven, 2001, 210-4. Sauaia, A. C. A.; Kallás, D. (2003) “Cooperate For Profits Or Compete For Market? Study Of Oligopolistic Pricing With A Business Game.” Developments In Business Simulation And Experiential Learning, Baltimore, V. 30, P. 232-242. Absel/2003. Shinohara, D. Y. (2003) Análise do EVA® como Instrumento de Gestão. VI Semead. São Paulo, 25 and 26 March, 2003. Silbiger, S. (1996) MBA em 10 Lições: As Mais Importantes Lições Das Melhores Faculdades De Administração Americanas. Rio De Janeiro: Campus, 1996. Stern Stewart. (1999) “ABC, The Balanced Scorecard And EVA: Distinguishing The Means From The End.” Evalua- tion, London, Volume One, Number Two, April 1999. Stevenson, W. J. (2001) Estatística Aplicada À Administração. Ed. 2001. São Paulo: Habra, 2001. Young, S. D.; O’byrne, S. F. (2001) EVA and Value-Based Management: A Practical Guide To Implementation. New York: Mcgraw-Hill, 2001. 248 Table of Contents Volume 31, 2004 Controlling the Complexity and Orenting Target Groups by a Modular, Server-Based Business Game System Learning Network Demonstration: Delivering Business Education in a Distance Learning Environment Economic Evolution, Human Capital Investment, and Adult Distributed Electronic Learning: A Literature Review Designing a Globalization Simulation to Teach Corporate Social Responsibility Developing and Teaching an Online / In-Class Hybrid: A Demonstration A Model for Evaluating Online Instruction An Evaluation of a Distributed Learning Course: A Students'-Eye Perspective Blended Learning Strategy Improved Business Writing Skills How to Receive and Process Attachemnts while Greatly Reducing the Risk of Viruses and Trojans Introducing Online Components to a Class: How to Increase teh Likelihood of Success Teaching Strategic Communications Online: Using Learning Outcomes to Develop a Case-Based Course Implementing Distance Approaches to Education: A Panel Discussion for ABSEL: Las Vegas, 2004 MANDI: Learning Management Through Field Sales Experience An International Capital budgeting Experiential Exercise A Primer To Combating Terrorism: Playing It Safe While On Overseas Assignment (An Experiential Exercise) Integrating The Business Curriculum With A Comprehensive Case Study: A Prototype The Case Brief: A Model For Case Analysis, Writing And Discussion Technology Infused Pedagogy And Delivery – A Sure Bet? A Proposal For Panel Discussion Absel Conference 2004 Research Strategy And The Bkl: Getting The Most From The Absel Archives Simple But Effective: Rediscovering The Class Discussion Needle And Thread: An Activity For Examining Various Management Behaviors A Customized Excel Data Analysis System For Use In Undergraduate Marketing Research Team Leader Selection - Does It Matter? The Power Of Perspective: Reframing Your Framing Skills For Innovative Instruction In Leadership And Influence Exercise: How Should Merit Raises Be Allocated? An Online Situation For Problem-Based Learning In A Junior-Level Management Course The Eden Alternative As A Roadway For Change: A Service Learning Quality Improvement Project Avoiding Catastrophe: The Role Of Individual Accountability In Team Effectiveness Omega Systems: A Change Management Exercise The Risks And Rewards Of Providing Students A Structured Cheating Opportunity Experimentation With Assessment Techniques: A Proposal For Panel Discussion Using A 2 - Page Case To Introduce Concepts Of Business Strategy Interactive Session The Integration Of Appreciative Inquiry And Experiential Learning For Peak Performance Appreciative Inquiry Case Story: New York City Leadership Challenge Individual Achievement Versus Team Performance: An Empirical Study With Business Games Some Strategists Don't Learn Or Can't Learn Computer Simulation: A Design Architectonic On The Value Of Bugs In Simulation Environments Online Sales Forecasting With The Multiple Regression Analysis Data Matrices Package Simulation Exercises And Problem Based Learning: Is There A Fit? A Study Of Business Game Stock Price Algorithms Assessing Individual Performance In A Total Enterprise Simulation Information Use In A Business Game Determining The Value Of A Firm Unsorting Algorithms For An Ordered List And Its Application To Business Simulations Teaching Public Finance Management Through Simulation Antecedents Of Game Performance Student Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Implementation And Impacts Of The Balanced Scorecard: An Experiment With Business Games Impact: Shocking The Legacy Mindset Implementation Of The Eepad Framework Of Business Processes In An Accounting Information Systems Course Are Business Games Really Delivering What Students Are Led To Believe?? Reporting Lessons Learned: What Gets Reported; Who Gains Value Teacher Expectations Of Classroom Teaching Practices In Developing And Presenting Course Information In Hong Kong Student Reactions To The Use Of A Computer-Based Simulation As An Integrating Mechanism For A Mba Curriculum A Cognitive Investigation Of The Internal Validity Of A Management Strategy Simulation Game The Casino Challenge: Making Simulation Delivery A Safe Bet! Accounting For Company Reputation: Variations On The Gold Standard Foreign Currency Hedging: A Simulation The Influence Of Variables Easily Controlled By The Instructor/Administrator On Simulation Outcomes: In Particular, The Variable, Reflection. Absel Awareness Among Business School Faculty Validating Business Simulations: Does High Market Share Lead To High Profitability? Simulation Debriefing Procedures Coaching And Business Simulations: A Formula For Success? A Seminal Inventory Of Basic Research Using Business Simulation Games The Influence Of Scorecard Evaluation On Decisions And Outcomes