CONGRUENCY OF CRITICAL PERFORMANCE FACTORS IN SIMULATED AND REAL WORLD ENVIRONMENTS: AN EXPLORATORY STUDY Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 11 CONGRUENCY OF CRITICAL PERFORMANCE FACTORS IN SIMULATED AND REAL WORLD ENVIRONMENTS: AN EXPLORATORY STUDY W. C. House, University of Arkansas H. S. Napier, University of Arkansas ABSTRACT A limited number of variables assumed to be of concern to Chief Executives in a manufacturing environment were chosen for analysis in simulated (The Executive Simulation) and actual companies. Return on Investment (ROI) was chosen as the dependent variable. Using multiple regression, four of six variables in simulated companies and three of four variables in the actual companies were found to be correlated with ROI. Comparisons and implications were discussed. INTRODUCTION In both real world and simulated business environments, hundreds of factors are present which exert influence on operating unit performance. Limitations on human time, effort, and attention that can be devoted to factor analysis make it unlikely that all of these factors can be examined in depth to determine what actions can be taken to improve performance. However, it may not be necessary to exhaustively examine all possible factors in countless combinations to determine which ones influence end results the most. In accordance with Pareto’s Principle (i.e.. a relatively few factors exert a disproportionate influence on final results), it seems quite likely that only a few factors will influence performance to a significant enough extent to warrant exhaustive evaluation. Rockart has identified for several selected representative industries, critical success factors covering a limited number of areas which if satisfactory will ensure successful competitive performance for the organization.1 From these critical success factors or performance measures, it is possible to determine significant information items needed to monitor performance in these areas. furthermore, it should be possible to determine which influencing factors exert more influence on key performance measures than others and hence deserve more attention, especially if the time and attention of the decision maker is limited. NATURE OF STUDY ENVIRONMENT This study of the effects of human and operational variables on organizational performance was focused initially on how performance by student teams in simulated business game environments was affected by selected critical performance factors. These factors were drawn from a list including factors which had been shown to exert some influence on organizational performance in past studies of business simulation results. The Executive Simulation, by Keys and Leftwich, was used to provide a suitable and representative simulated business environment for the study. In the Executive Simulation, up to six teams in each industry make pricing, advertising, research and development, product mix, production scheduling, salesmen compensation, and financing decisions which influence sales, profit, and asset performance along with external economic conditions. The simulated business environment is designed to represent the operations of a group of consumer goods manufacturing companies. Seven variables were chosen by analysis and examination in the study and included marketing, finance, operations, and personnel factors. These variables represent items of concern to chief executives in a changing manufacturing environment. Return on investment was chosen as the dependent variable since it is widely accepted as a performance measure and is easy to calculate and understand. Each company in the study was compared to its next listed competitor. For example, company one was compared to company two, company two to company three, and so on for the twelve companies considered in the two industries represented. A value of one was assigned if the selected company had a larger return on investment value than its competitor and 0 if the value was the same or less. In this way, the relationship of the values assumed by selected variables to higher or lower return on investment figures can be explored In an approximate manner. In order to allow for the lagged influence of the independent variables upon the dependent variable, return on investment for the eighth quarter was used as the dependent variable and all of the independent variables were measured at the end of the fourth quarter. The independent variables included those relating to the personnel, marketing, innovation, and finance areas. High morale was selected as a measure of the importance attached by top management to the human resource. If high morale was chosen as an important goal for managers, a one was assigned; if not, a zero value was included. Market share was selected as a measure of marketing effectiveness. Thus, a one or a zero was assigned to this value depending on whether the firm had a larger share of the market for product “A”, the standard consumer product offered by all companies. Advertising expenditures divided by past years sales revenues was used as a second measure of marketing effectiveness, again receiving a one or zero value depending on whether the advertising/sales ratio was higher than that of its competitor. In a similar fashion, research and development outlays divided by sales revenues was used as an indication of efforts to generate new products and to improve production processes. One was assigned if R&D$/Sales$ was higher for a given company than the same ratio for its competitor and zero if not. Two financial factors were also included: Cash on hand at the end of quarter four and fixed asset values at the end of quarter four. If the variable value was greater than the previous period (quarter three), a value of one was assigned and if the same or lower, a value of zero was recorded. In summary, all variables were assigned values of zero or one and variables included were selected from a list of variables believed to be important in their effects on total company performance. The dependent variable, Return on Investment, was measured at the end of two years of play while the Independent variables were measured at the end of the first year of play. The investigation was made to determine if company performance in the form of return on investment could be predicted accurately one year in advance and to indicate which ties from among a selected group of performance factors Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 12 had the most significant influence on the criterion variable. STUDY RESULTS SHOW FOUR OF SIX VARIABLES ARE HIGHLY CORRELATED TO RETURN ON INVESTMENT IN SIMULATED ENVIRONMENT The use of a stepwise regression program allowed determination of which independent variables when combined showed the highest correlation with return on investment. Plant value showed less correlation with the criterion variable than any of the other independent variables, so it was dropped from consideration. Advertising/sales dollars also did not exhibit a high degree of correlation with the selected performance measure and it was dropped out during the second round of correlation with the selected performance measure and it was dropped out during the second round of correlation calculations. The four remaining variables--market share (Product A), R&D Outlays/Sales Revenues, High Morale, and Cash Position--turned out to be significantly correlated to return on investment. The multiple correlation coefficient was .87, with .75 percent of the variation in ROI being explained by changes in the four selected variables. From the computations, the following multiple regression equation was developed: Yc .38 + .61X7 - .29X5 - .31x3 + .28X6 The standard error of estimate was .32 and the computed F value was 5.35 compared to a table value of 4.12, indicating the results are significant at the .05 level. For the twelve companies considered in the study, the multiple regression equation predicted accurately in twelve of twelve cases whether the return on investment of a given company will be higher or lower than that of its next listed competitor. The study also suggests that higher rates of return on investment are likely to result if preceding periods Market shares, R&D Outlays/Sales Revenues and Cash positions are higher than comparable values for competitors. The fourth factor, high morale, also suggests that teams which place less importance on human resources than other factors are likely to achieve higher financial performance than their competitors. In other words, they are more production oriented than personnel oriented in their decisions. THREE OF FOUR VARIABLES HIGHLY CORRELATED TO RETURN ON INVESTMENT IN STUDY OF ACTUAL COMPANIES FROM SELECTED INDUSTRIES If business simulation environments are reasonably representative of real world situations, there should be some degree of congruence between critical performance factors found to be important to higher levels of performance in simulated environments on a logically consistent and intuitive basis with those attained in actual company and industry situations. However, few studies have been conducted thus far which confirm or deny this hypothesis. In an earlier study, House and Napier showed that for twenty-two companies in eleven industry groups which were paired according to high and low innovation ratings received in a FOR-TUNE Corporate Reputation study, a three variable analysis resulted in predicting whether a company would have a higher innovation rating than its competitor in 20 of 22 cases.3 Using R&D expenses! Sales revenues (This Year versus Last Year), R&D expenses (Company versus its Competitor), and Return on Assets (Company versus Competitor), a multiple correlation coefficient of .89 results which is significant at an .05 level. Typically, companies with lower innovation ratings did not increase their R&D expenditures from year to year and tended to have R&D expenditures and rates of return which were lower than those of their competition. To provide a basis for comparison with critical performance factors identified and evaluated in the simulated business environment, sixteen companies in eight industry groups were selected, for which data were available in The Value Line Investment Survey analysis, as to the importance of critical performance factors in influencing operating results. Return on total capital for 1983 was used as the dependent variable and independent variables selected included 1982 Cash levels, R&D Outlays/Sales Revenues for 1982 and Attracting/Holding Talented personnel (1983 Corporate Reputation Survey-- Fortune) as a measure of human factors. Two companies were selected from each of eight industry groups, including food processing, paper, oil, electronics, electrical equipment, aerospace, pharmaceutical, and computing equipment. A one or zero entry was made for each of the companies, depending on whether it had higher (=1) or lower (=0) Cash levels, R&D Expenses/Sales Revenues, Sales Revenues and Attracting/ Holding Talented Personnel than its next listed competitor. The variable value for Return on Total Capital was also entered as a one or zero, depending on whether that value was higher or lower than that of its selected competitor. When a stepwise regression program was utilized to correlate the dependent and independent variables, three of the four variables-Attracting/Holding Talented Personnel, R&D Expenses/Sales Revenue, and Sales for Selected Company (versus its nearest competitor) produced a multiple correlation coefficient of .80 with a standard error of estimate of .34. The computed F value was 7.50 versus 3.49 which is significant at the .05 level. The fourth variable, Cash level versus competitor did not add significantly to the multiple correlation results. Approximately sixty-five percent of the change in the dependent variable was explained by changes in three of the four dependent variables. From the regression computations, the following multiple regression equation was developed. Yc = .18 + .72X4 - .22X2 + .11X3 For the sixteen actual companies considered in this study, the multiple regression equation correctly predicted whether the Return on Total Capital will be higher than that of its competitors in fourteen of sixteen cases. As Table II indicates, only in the case of the two pharmaceutical companies does the multiple regression equation fail to predict correctly a higher or lower Return on Total Capital for a given company than for its competitor. The results also indicate that higher rates of return on Total Capital are likely to result if preceding periods Market Share (i.e., sales revenues versus competitors), R&D Outlays as a percent of Sales Revenues, and Ability to Attract and Hold Talented Personnel are higher than comparable values for their competitors. IMPLICATIONS FOR FURTHER RESEARCH AND INVESTIGATION In the simulated environment, four of the six selected variables, including Market Share, R&D Outlays/Sales revenues, High Morale, and Cash Levels (Year One) Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 13 appeared to be related to Return on Investment. Advertising Expense/Sales Revenue and Plant Value did not add significantly to the explained variance. In the case of the actual companies from selected industries, three of four variables examined--Market Share, R&D Outlays/Sales Revenues, and Ability to Attract/Hold Talented Personnel-- appeared to relate to the Return on Total Capital for the next period. Unlike in the simulated environment, Cash levels was not included as a significant variable in the actual company multiple regression equation. This result may be partially explained by the fact that large Fortune 500 companies are not normally subjected to the same cash constraints as companies in smaller annual sales categories. The study suffers from several significant limitations. Only a few companies in selected industries were considered in both the simulated and actual environments. Likewise, only a few of the many possible important variables affecting return on investment could be considered. Not all the variables considered in both studies are strictly comparable although there is a rough coincidence. A stronger correlation occurs in the simulated environment with four of six variables than in the actual environment with three of four variables. Despite these limitations, there is considerable evidence that a selected few variables significantly influence key performance measures and if these can be identified and analyzed for a cross section of actual industries, it may be possible to achieve greater congruence between key performance indicators in actual and simulated business environments. Collection of additional data on a larger number of variables over longer time periods for more companies in both simulated and actual business environments should provide a more accurate assessment of which performance variables are most significant in influencing total company performance. In addition, in the simulated environment the personnel variable turned out to be negatively related to return on investment, suggesting some incongruency in importance attached to this factor in the two environments. With additional, expanded studies over longer periods of time. it should be possible to identify performance variables most significantly affect overall company performance in actual business environments and to determine to what extent these factors may vary with the size of the company and nature of the industry. Comparisons of performance factors in simulated and actual environments should help to determine how well simulated should be made to business simulations to make them more realistic. In actual company situations, identification of key performance variables could assist managers in achieving better performance while minimizing expenditures of time and effort and indicating factors which information systems should report on a regular basis REFERENCES [1] John F. Rockart, “Chief Executives Define Their Own Data Needs,” Harvard Business Review (March-April 1979), pp. 81-93. [2] Nancy Perry, “America’s Most Admired Corporations,” Fortune (January 9, 1984). [3] N. C. House and Herman S. Napier, “Innovation and Management: Gaining an Edge on the Future,” Proceedings, Southwest Business Symposium, Edmond, Oklahoma (April 6, 1964). [4] “The U.S. Still Leads the World in R&D Spending,” Business Week (June 20, 1983). Table of Contents Volume 12, 1985 One Step Beyond Naiveté: Laboratory Simulation of Strategic Management Predicting Performance Over the Course of the Simulation Congruency of Critical Performance Factors in Simulated and Real World Environments: An Exploratory Study Stimulating Learning and Creating Course Involvement Through the Use of Debates A Novel Approach to Writing Texts ABSEL Revisited: In-Class Student Involvement Increases Learning Bus-Sim Micro: An Introductory Business Simulation Demonstration Solutions Diversified: And Experiential Introduction to Management Information Systems Simulating Rigorous Analysis in Stimulation Gaming The Use of Decision Support Systems (DSS) and Operations Research/Management Science (OR/MS) Techniques to Enhance the Learning Experience of Students Participating in Computerized Simulation Problems Associated with the Administrative Use of Student Ratings Enthusiasm Awareness in the Experiential Classroom Participation Expectations of Students in Experiential Settings Using the Computer as a Planning Aid in an Applied Marketing Course Integrating Microcomputers in the Marketing Curriculum Through the Use of Marketing COMPUPROBS Giving Students Experience in Team Report Writing Description and Results of an Experiential Approach in the Teaching of Business Policy Using Real World Experiences as an Experiential Activity: the Page Technique A Guide to Writing Microcomputer Simulations The Development of a Microcomputer Distributed Processing Business Game A Monte-Carlo Approach to Interactive Gaming Single Case Experimental Design as an Alternative to the Conventional Approach Learning Theory and Research Design: How Has ABSEL Fared? Criticisms of the Use of Simulations in Economics: A Rebuttal Using the General Management Game in the Business Policy Course General Management Policy Simulations: Which Games are Popular, and Why do Professors Stop Using Games? Business Policy Content and Teaching Methods: Comparisons of the Hegarty and Summers and Boyd Findings Let's Talk to the Experiential Authors A Computerized Logistics Game for Micros Effects of Contextual Factors on Bargaining Decisions of Experiential Group Representatives The Linking of a Collective Bargaining Simulation and a Business Simulation Game - A Case Study Students are Surrogates in Arbitration Studies AIRWAYS: A Microcomputer Simulation of a Service Industry A Proposed Interactive Inventory Control Simulation Microcomputer Demonstration Exercises for a Course on Creativity and Problem Solving Processing Skills for Teachers, Trainers and Managers Developing the Competencies of Resistance to Stress and Accurate Self-Assessment Raising Elizabeth: Socializing Girls to Aspire to Careers as Managers, Secretaries or Corporate Wives Factors Influencing Perception: Gaining Understanding Through Experience Painless Computer Simulation The Effects of Differential Grade Weights on Business Game Learning levels An Investigation of the Validity of a Recommendation for Experiential Exercise Debriefing Simulation and Experiential Practices of Faculty: A Data-Based Workshop The Live Case Study: Filling the Gap Between the Case Study and the Experiential Exercise A Reactive Microcase Pedagogy Advantages of a Multigame Simulation Course Using the Logistic Function to Score Computer-Supported Games A Simulation Model to Calculate Session Time for Running Simulation Models in a Shared Resource Environment On the Core Size of Small Task Groups and the Cause for the Core Size to Emerge