THE IMPACT OF DECISION SUPPORT SYSTEMS ON THE EFFECTIVENESS OF SMALL GROUP DECISIONS - AN EXPLORATORY STUDY Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 132 THE IMPACT OF DECISION SUPPORT SYSTEMS ON THE EFFECTIVENESS OF SMALL GROUP DECISIONS - AN EXPLORATORY STUDY John F. Affisco, Hofstra University Michael N. Chanin, Baruch College, CUNY ABSTRACT This paper presents the results of an empirical investigation of the impact of Decision Support Systems on small group decision effectiveness. The study is conducted in the context of the Business Management Laboratory. Wolfe’s Business Management Laboratory Decision Support System (BMLDSS) was utilized to aid decision making by all treatment groups. Results indicate use of the BMLDSS did not lead to significantly superior performance as measured by financial performance variables. Further, no significant increase in satisfaction with or confidence in the decision making process could be attributed to the use of the BMLDSS. INTRODUCTION The last decade has been one of continuing development for Decision Support Systems. In their seminal work of 1978, Keen and Scott Norton (1978) forecasted the following accomplishments for Decision Support Systems (DSS): (1) The impact is on decisions in which there is sufficient structure for computer and analytic aids to be of value but where managers’ judgement is essential; (ii) The payoff is in extending the range and capability of managers’ decision processes to help them improve their effectiveness; (iii) The relevance for managers is the creation of a supportive tool, under their own control, which does not attempt to automate the decision process, predefine objectives, or impose solutions. Thus, a basic premise of DSS is that they can improve a decision maker’s effectiveness. Implicit in this line of reasoning is the assumption that the product of man- computer interaction will exceed the sum of its component parts. That is, the ran-computer partnership will think and process information in a way not approached by the man or the computer alone. Ten years after this work ye find ourselves in an environment of systems proliferation due, in large part, to the development of the personal computer and significant advances in communications technology. In a majority of today’s organizations the concept of a centralized information systems function is changing rapidly. The diffusion of technology throughout the organization is creating a phenomenon known as end user computing. That is, applications are being developed, used and maintained by managers and staff members. Indeed, some experts believe that we are in a period of transition between the days when no one in the organization but the information systems group had access to the computer to the day when everyone will. Many of these end user systems are of the decision support variety. Given this reality, an important issue for investigation is whether or not any evidence exists to conclude hat the use of DSS does improve decision makers’ effectiveness. Empirical studies directed at determining the impact of man-machine systems on decision processes have appeared in the literature for a number of years. For example, some representative studies include Benbasat and Dexter (1982), Kasper (1985), Goslar, Green, and Hughes (1986), and Cats-Baril and Huber (19271. Benbasat and Dexter (1982) utilized a multiperiod production decision making game to investigate individual differences in the use of decision support aids. Specifically the study was designed to assess whether decision aids can improve the performance of low analytics in task environments unsuitable for their cognitive style. Subjects were divided into high and by analytic cognitive styles based on scores on the Group Embedded Figures Test. For each cognitive style half of the subjects used a decision aid while the remaining half did not. Results of the study provide support for the notion that appropriate information system design can help overcome a mismatch between task environment and psychological type. Kasper (19851 performed an empirical investigation of whether a DSS can improve a decision maker’s effectiveness. This free simulation study was conducted in the BML/SLIM environment. In addition to deciding on a subset of thirteen variables each quarter, participants were required to forecast firm’s return on assets, market share, and stock price. The absolute difference between these anticipated results and the subject’s actual outcomes served as the dependent variables for the study. The subjects were divided into two groups: the non-modeler group would be alloyed to use the SLIM DSS but would not be able to save their work for future use; the modeler group would be able to save their applications. The findings include the facts that the DSS modeling group outperformed the non-modelers on forecast accuracy and that there was no significant difference in system use between the two groups. Goslar, Green, and Hughes (1986) conducted a laboratory study aimed at determining how a specific DSS can assist decision makers in dealing with ill-structured problems. Forty-three sales and marketing personnel of nineteen middle-to-large regional and national business organizations were subjects of the study. A case vas utilized to provide a realistic, ill-structured marketing problem situation. Subjects were randomly divided into six groups which received treatments based on three independent variables; DSS availability, DSS training, and data level. Contact with the DSS vas through trained intermediaries who developed requested applications using IFPS, More of the factors DSS availability, DSS training, or data level significantly affected subjects’ decision making confidence, decision making processes, or performance Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 133 levels. Cats-Baril and Huber (1987) also conducted an empirical study of the use of DSS for ill-structured problems. Senior business students were faced with a career planning task in which they were required to generate a list of career objectives and prioritize them, develop a list of strategies that would allow them to achieve the most important objectives, and, finally, prioritize these strategies. The study investigated six experimental configurations created by manipulating three system-design characteristics: heuristics, interaction, and computerization. Quality of performance, productivity, satisfaction, and decision making confidence were measured for each configuration. The major finding of interest is that while the use of heuristics led to better performance, this was so whether the heuristics were implemented by paper and pencil or by computer. In an article reviewing the present state of DSS research Henderson (1987) concludes that the vast majority 3f applications research has concentrated on individual decision makers dealing with a single specific task. He concludes that their remains a need for additional empirical studies aimed at determining the effectiveness of DSS and improving existing implementation methodology. A further conclusion is that research directed at small group decision making should be encouraged since little has been done in this area. This study begins to extend the empirical research by focusing on repetitive decision making by small groups. Specifically we utilize a free simulation approach to examine whether a specific DSS can lead to improved small group decision making performance. The free simulation environment allows for the investigation of repetitive multiple decisions over an extended time horizon. RESEARCH METHODOLOGY This research was conducted in the context of the Business Management Laboratory Simulation Game of Jensen & Cherrington (1984). This is a complex business game in which approximately thirty five decisions are made for every quarter of play in the version used. Specifically, two products were sold in a single market area, The BML has been successfully utilized as a research environment for studies of decision making previously (See, for example. Affisco & Chanin 1988, 1987, 1986) and Courtney, DeSanctis, and Kasper (1983)). Twenty groups of senior business students in the required capstone Business Policy course during the Summer 1988 session were the subjects of the study. These groups were generally constructed of four members with different majors (i.e. Accountancy, Finance, Marketing, Management, etc. ) within constraints imposed by the registration process. The groups were randomly divided into two six team and one eight team industries across two course sections. Thus, in each section there were three teams from each of the two six team industries and four teams from the eight team industry, essentially composing control and treatment sections. Each of the control groups made eight quarterly game decisions in class under the observation of the instructor. These decisions were to be made without the use of any Decision Support System or any other computer support. Control group members were allowed to use calculators as a computation aid. In • like manner, treatment groups made the first four quarterly game decisions. After the first year vas completed the treatment groups were required to use a DSS to support their decision making process. Specifically, Wolfe’s (1988) Business Management Laboratory Decision Support System (BMLDSS) was employed. The BMLDSS is a library of eight Lotus 123 spreadsheet templates. Each module may be operated in one of two modes. The Conversational Node Simulates a conversation with an expert in the particular decision making area being addressed. The Data Entry Mode allows simple data entry directly to the program bypassing the opportunity for expect consultation. The eight modules that compose the library consist of: 1. Long-term Demand Forecasting - Calculates long- term demand y product and market area based on economic, seasonal, and endogenous factors; 2. Short-term Demand Forecasting - Forecasts firms quarterly demand based n marketing variables. Considers impact of competitors, and derives optimistic, conservative and most likely scenarios. In addition, this module assists in calculating quarterly production capacity requirements; 3. Short-term Raw Material Requirements Derives quarterly raw material requirements, costs alternative purchase policies for two quarters; 4. Operations Manager - Schedules production. suggests maintenance budgets, costs alternative production schedules, evaluates alternative inventory accounting methods, and derives cash disbursements based on the production variables; 5. Bond Offerings - Simulates the bond market and determines whether an offering is acceptable, what the net proceeds and effective interest rate will be. Also evaluates bond call backs; 6. Stock Offerings - Simulates the underwriter and determines whether the offering is acceptable, determines the cash proceeds of the issue, and its effect on the firm’s most recent stock price; 7. Cash Flow Forecast s Creates a cash flow analysis based on estimated operations for the current quarter; 8. Pro Forma Income Statement e Creates trial income statements based on estimated operations for the current quarter. A complete description of the BMLDSS was distributed to each student in the treatment groups. Each group was trained in the use of the programs by an experienced Doctoral Research Assistant. In addition, time was made available for hands on practice with the system for each group. For all treatment groups the second year decision making sessions were conducted in a laboratory equipped with a local area network with four IBM PC/XT computers serving as work stations. However, during this research these PCs were operated as stand alone machines. Thus, during each decision making session time on these four machines vas equally divided among the ten groups. To assure the use of the DSS, treatment groups were required to hand in printed output with each of their second year Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 134 decisions. Actual game performance data were used to determine the impact of DSS (the treatment) on decision making effectiveness. Specifically, financial measures at the end of second year were compared with the identical measures for the first year. The average differences for the control and treatment groups were compared to determine if any significant differences existed. Thus, a pretest/posttest with control experimental design was utilized In addition, structured questionnaires were administered during a debriefing session to all groups at the end of each year. For the first year, the questionnaires were identical for treatment and control groups. The questionnaires for the second administration were group specific since certain questions referred to the use of the DSS in the decision making process. These questionnaires were aimed at determining the groups satisfaction with the decision making process, their confidence in the resulting decisions, the number of alternative decision sets that were evaluated before arriving at the submitted decision and the amount of data that was considered In making the decisions. RESULTS AND DISCUSSION Decision effectiveness related to the use of the DSS was tested by comparing average differences in financial performance variables over the two year horizon. Specifically, mean differences and the corresponding standard deviation were calculated for control and treatment groups for ROI, Stock Price, Credit Rating, Earnings Per Share, Total Sales Dollars, at. Unit Manufacturing Con for each product. The results are presented in Table 1. For three of the variables, Stack Price and both unit costs, DSS groups improved their performance by a greater amount than the control groups. For the four remaining variables, ROI, Credit Rating, Earnings Per Share, and Total Sales Dollars, control groups shoved superior performance improvement. To determine if these results were significant a series of t-tests were performed. The results of these tests, which are also presented in Table 1, indicated that no significant difference in performance could be attributed to the use of the DSS. Satisfaction with the decision making process and confidence in the resulting decisions were measured by a set of five point Likert-scale questions similar to those presented in Cats-Baril and Huber (1987). The mean individual scores for satisfaction and confidence are presented in Table 2. The satisfaction scores increased for both DSS and Non-DSS groups over the two years. However, t-tests indicated that the improvement was not significant at the 0.05 level. Similarly, confidence scores increased for both sets of groups from year 1 to year 2. Once again, these differences were shown to be insignificant on the basic of the results of t-tests. Additional t-tests were performed on the changes between scores over the two years. These test also resulted in insignificant differences between the groups. Data with respect to the number of decision alternatives considered and the amount of data utilized in the decision making process were also collected. In both cases no difference was observed over the two years for either the DSS or Non-DSS groups. In general, the findings of this research provide no support for the premise that the use of Decision Support Systems improves small group decision making effectiveness. Although this contradicts the conventional wisdom, it is in the mainstream of findings of existing research studies that focus on individual decision makers. Several factors could explain these results. Due to the limited number of computers available, each treatment group was allocated approximately thirty minutes of computer time during decision making sessions. During debriefing sessions after the second year all the treatment groups indicated that the amount of available computer time was inadequate. Further, many subjects indicated that much of the time was spent dealing with the mechanics of the software rather than in decision making. A more encompassing issue here is the level of training in the basic software, Lotus 120. Most of the subjects had received only cursory exposure to Lotus 123 in a Data Processing course early in their college studies. Few were required to use and further this knowledge in the business courses that followed. Also with respect to computer use, giver the ready availability of personal computers and software outside the laboratory it was extremely difficult to control their use by the Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 135 control groups, and by the treatment groups during the first year. Another factor of interest is that the treatment groups were expected to trust implicitly in results generated by the DSS which utilized models of which they had no knowledge. The subjects received no instruction in the models that were utilized nor any information as to there sensitivity or limitations. This may account for the fact that many subjects felt that the BMLDSS was complex and not easy to comprehend. The fact that the BMLDSS was designed to support two market areas while only a single market area was in play might also account for this result. Although technical training in the use of BMLDSS was given to the subjects, no training in its use as part of a logical decision making process was offered. That is, the training consisted solely of how to operate the DSS. The fact that use of the BMLDSS did not significantly affect group decision making performance suggests that future training efforts may need to include concentrations in logic and decision making processes, as well as technical instruction if users are to take full advantage of DE capabilities. The findings here are similar to those of Goslar, Green and Hughes (1986). Finally, while this research deals with small groups, the BMLDSS is not a group decision support system. Thus, while it provides a modelling and calculation capability, no provision is made for assisting in the basic processes involved in group decision making. This raises the question that perhaps the use of BMLDSS in concert with a specific problem solving technology (for example, Nominal Group Technique or Dialectical Problem Solving Technology) may lead t3 improved results. CONCLUSION This free simulation study investigated whether the use of a modelling and calculation oriented Decision Support System could improve small group decision making effectiveness. Findings, based on both financial performance variables and behavioral variables, indicated that the use of the BMLDSS had no impact on decision making effectiveness. These findings add to the very mixed results of empirical studies of DSS effectiveness that may be found in the literature. Clearly additional studies focused on issues such as technical training, use of structured decision making process, and groups effects are necessary to add to our knowledge of the effectiveness of such Decision Support Systems. REFERENCES Affisco, J.F. & Chanin, M.N. (1988), “An Empirical Investigation of Integrated Spatial-Proximity MCDM Behavioral Problem Solving Technology Group Decision Models”, in P. Sanders & T. Fray (Eds.), Developments in Business Simulation & Experiential Exercises. Stillwater, OK: Association for Business Simulation and Experiential Learning, 56-60. Affisco, J.F. & Chanin, M.N. (1987),”‘Simulation Gaming as an Experiential Context for the Study of Multicriteria Decision Making”, in L. Kelly & P. Sanders (Eds.), Developments in Business Simulation & Experiential Exercises, Stillwater, OK: Association for Business Simulation and Experiential Learning, 1-5. Affisco, J.F. & Chanin, M.N. (1986), “A Study of the Effectiveness of Assumption Making in Strategic Planning”, Proceedings of the Northeast American Institute for Decision Sciences, 91-93. Benbasat, I. & Dexter, A.S. (1982), “Individual Differences in the Use of Decision Support Aids”, Journal of Accounting Research. 20(No. 1), 1-11. Cats-Baril, W.L. & Huber, G.F. (1987), “Decision Support Systems for Ill-Structured Problems: An Empirical Study”, Decision Sciences. 18(No. 3), 50 372. Courtney, J. F., DeSanctis, G. & Kasper, C. N. (19 3), “Continuity in MIS/DSS Laboratory Research: The Case for a Common Gaming Simulator”, Decision Sciences, 14(No. 3), 419-439. Goslar, M.D., Green, G.I., & Hughes, T.H. (1986) “Decision Support Systems: An Empirical Assessment for Decision Making”, Decision Sciences, 17(No. 1), 79-9 1. Henderson, J.C. (198 , “Finding Synergy Between Decision Support Systems and Expert Systems Research”, Decision Sciences, 19 No. 3), 333-349. Jensen, R. & Cherrington, D.J. (1984), The Business Management Laboratory, 3rd. Ed., Plano, TX: BPI. Kasper, G.M. (1985), “The Effect of User Developed DSS Applications on Forecasting Decision-flaking Performance in an Experimental Setting”. Journal of Management Information Systems, II(No. 2), 26-39. Keen, P.G.W. & Scott Morton, M.S. (1978), Decision Support Systems-An Organizational Perspective Reading, NA; Addison-Wesley. Wolfe, J. (1988), The Business Management Laboratory Decision Support System, Tulsa OK: Table of Contents Volume 16, 1989 Quality Control Circles (QC™s): Towards a Computerized Simulation The Canadian Hospital Executive Simulation System (CHESS) The Impact of Using Group Performance Evaluation as an Experiential Exercise The Impact of Leader and Team Member Characteristics Upon Simulation Performance: A Start-Up Study Planning for Career Success: Is Where you are Going Where you Really Want to Be? The Production Frontier: Modeling Production in the Computerized business Simulation A Study of the Need for Valid Business Game Algorithms Modeling the Human Component of Business Simulations A Stimulating Simulation in International Business Business Ethics, Experiential Exercises and Simulation Games Collective Bargaining Simulation: Adding Reality Through Point Scoring The Use of Experiential Teaching Techniques: Creativity vs. Conformity Visualization and Guided Imagery in the Organization Behavior Class: An Experiential Exploratory Approach Arranging an Agenda: An Activity on Running Better Meetings Harried Harry: An Experiential Capstone for Students of Organizational Behavior Coping with Stress: An Experiential Exercise Fairness in the Classroom: An Empirical Extension of the Notion of Organizational Justice A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation Competency Based Development: A Management Development Exercise Simulation Performance Revisited: The Fit Between Instructor Style and Learning Style An Evaluation and Application of an Instrument for Measuring Pedagogical Effectiveness A Knowledge Based System to Support Reasoning by Analogy for Business Simulation Gaming using Forecasting Accuracy as a Measure of Success in Business Simulations The Development of Algorithmic Functional Business Games Strategy Design, Process and Implementation in an Unstable/Complex Environment: A Second Exploratory Study Simulation Integration Contrasts Between MBAs and Undergraduates in the Capstone Policy Course An Investigation of the Real World Usefulness of a Strategy and Policy Course Using a Business Simulation Framework Duel (sic) Views of Internships, as Experiential Learning The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - An Exploratory Study An Investigation of the Relationship Between Formal Planning and Simulation Team Performance Under Changing Environmental conditions Sensitivity Analysis with the Complete IFPS/Personal Student Analysis Package: A marketing Decision Support System A New Approach to Teaching Salesmanship using Persona, Microskills, and a Sales Process A Rational Case for Synthetic Experience as a Prime Ingredient in the Marketing Curriculum SalesHire: A Microcomputer-Based Salesperson Selection Exercise TRANSECON: An Interactive Program for Learning Transportation Economics Hypercard as a Construction Tool for Short Instructional Exercises A Game to Introduce Accounting Information Systems Students to Certain Internal Control Concepts "Commitments" - A Demonstration Proposal An Analysis of Popular Games as Experiential Models for Corporate and Collegiate Management Education An Exploratory Study of the Effects of Strategic Emphasis in Management Games on Attitudes, Interest, and learning in the Business Policy Course Predicting Individual Decision Making Performance in a Business Simulation: An Empirical Study Strategic Planning And Organizational Performance In A Business Simulation: An Empirical Study, PAM (Planning Action Management) Simulation of a District Sales Territory Lifelong Learning and ABSEL: An Inquiry on Definitions and Relationships A Review of Salient Trends in Proceedings: A Fifteen year (1974 - 1988) Review of ABSEL Contributorship