LEARNING THEORY AND RESEARCH DESIGN: HOW HAS ABSEL FARED? Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 86 LEARNING THEORY AND RESEARCH DESIGN: HOW HAS ABSEL FARED? Richard J. Butler, Rochester Institute of Technology Peter M. Markulis, SUNY Geneseo Daniel R. Strang, SUNY Geneseo ABSTRACT Since its inception in the early 1970’s, a major focus of ABSEL has been research on learning. However, after years of ABSEL research, many of us perceive little in the way of definitive findings relative to understanding the relationship between experiential exercises and learning outcomes. This paper examines two major problems that may have impeded our work in the area: lack of adequate research designs and lack of a good framework for conceptualizing the learning process. The authors have reviewed eight years of ABSEL proceedings and classified studies of learning according to their use of good research design techniques and according to the nature of the learning that the researchers investigated. Suggestions for altering research methods and foci of study are offered. INTRODUCTION One has only to peruse any one of the Annual Proceedings or any of the programs for the annual meetings to note that ABSEL members have been preoccupied, since the inception of the Association, with the issue of learning attributable to simulations and other experiential exercises. Yet little has emerged from this collective effort that establishes what learning outcomes occur with experiential exercises and under what conditions. The project reported was undertaken for three reasons. First, to review what ABSEL researchers have done in terms of linking simulations/experiential exercises and learning theory. After reviewing the research studies in the ABSEL Proceedings it is our contention that most attempts to show that learning is associated with simulations have fallen short of the mark. The difficulty of developing such evidence takes on special significance for ABSEL given the interest in simulation/experiential exercise pedagogy on the part of much of the membership. Second, we are interested in identifying the causes of this state of affairs (i.e., lack of definitive evidence) . Two factors in particular seem to have impeded development of knowledge in this area: inadequate attention to research design and the lack of a paradigm to guide investigation of learning. A third for this project is to suggest the value (for future researchers in this area) of a standard taxonomy for categorizing and classifying learning outcomes, that is both consistent with generally accepted learning theory and relevant to the objectives and purported benefits of simulations/experiential exercises. Some earlier attempts at evaluating the state of ABSEL research have been undertaken by ABSEL researchers. For example, Wolfe (1981) reports that a review of ABSELs Proceedings for the years 1976-1980 revealed: “a proliferation.., of ‘pre-experimental’ designs and an almost total absence of true experimental studies ... All the studies appearing in the ABSEL Proceeding failed to meet the criteria of external validity, and more importantly ...very few ... met the criteria for internal validity.” (p. 72) In an earlier article Wolfe (1976) noted that even the most rigorous designs (viz., Stanley and Campbell’s separate- sample pretest-posttest control group) had not been fully implemented by ABSEL researchers. Keys (1976), in a review article on simulation gaming and learning, located only eight studies (mostly non- ABSEL) ‘that utilized definite criteria for the measurement of learning and professionally acceptable research criteria” (p. 173) . Furthermore, the focus of the research in these studies appeared to be on pragmatic tests of the efficacy of simulation-gaming vis-à-vis other teaching techniques. There is little indication that any paradigm of learning guided the selection of outcome measures. It is also the case that at least a few ABSEL researchers have recognized the need for a paradigm that could be used to guide research on learning outcomes (Brenenstuhl and Catalanello, l976~ Burns and Gentry, 1977; Gentry and Burns, 1981) . Perhaps the most sophisticated design was suggested by Brenenstuhl and Catalanello, one which incorporated learning theory and research methodology to gather data on different types of learning outcomes (e.g., cognitive development, skills, motivation, and satisfaction) , and correlate them with personality variables. Certainly in terms on conceptualization, this design represents a strong proposal, which attempts to capture very different kinds of learning outcomes using a rigorous experimental methodology. In another example, Gentry and Burns (1981) propose operationalizing learning (one of two dependent variables in their model of effectiveness of experiential exercises) using Blooms Taxonomy of cognitive outcomes. Smith (1981) issued a call for game designers to incorporate the implications of Blooms Taxonomy and other learning models into their simulations. It is evident from the above that the issues of research design and recognition of the need for a paradigm of learning outcomes to guide researchers are not entirely missing from ABSEL. However, it appears clear, as will become evident below, that much more needs to be done by us as an association, that an occasional exhortation to strengthen our research designs or a passing reference to the value of employing a shared paradigm of learning. Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 87 A TAXONOMY OF EDUCATIONAL OBJECTIVES Basically, a taxonomy is a way to classify or categorize according to natural relationships. A taxonomy must be constructed so that the order of the terms corresponds to some ‘real’ order among the phenomena represented. A taxonomy is helpful to: (1) standardize communication; (2) focus attention on under researched areas; and (3) classify phenomena (the most basic level of science). Characteristics of a good educational taxonomy are that it: (1) actually improves communication; (2) is logical, consistent and easy to use; and (3) is consistent with relevant and accepted psychological principles (Bloom, 1956) In the education literature that has come to be known as Blooms Taxonomy of educational objectives was first introduced in 1956 (Krathwohl, Bloom, and Masia). This taxonomy is considered a standard way to classify learning objectives and outcomes and has been successfully employed by researchers in the field of education. It classifies learning outcomes into three domains: (1) cognitive (or knowing); (2) affective (or feelings); and (3) psychomotor (or doing). The cognitive domain deals with recall and recognition of knowledge and with the development of intellectual abilities and skill. An example of cognitive skills emerging from a simulation experience might be the ability to forecast sales given information on historical sales patterns, the state of the economy, and competitors * strategies. The affective domain refers to the way and degree to which learners are sensitized to learning. The affective domain emphasizes a feeling, tone, or a degree of acceptance or rejection of learning. In a simulation context, the affective domain might refer to the goal of having students enjoy, and perceive value in managing a (simulated) firm. The psychomotor domain refers to the development of motor skills. An example of a simulation aimed at the psychomotor domain is the flight simulation, used by the airlines to train pilots, demonstrated at the 1983 ABSEL Conference at Tulsa, Oklahoma. Within two of the three domains, Bloom, et. al., have described several levels. In the cognitive domain, the most basic level is knowledge (of specifics, terminology, facts; of ways and means of dealing with specifics; of universals and abstractions). The second level is intellectual abilities and skills, principally comprehension. It includes the ability to translate, interpret, and extrapolate. The third level in the taxonomy is the ability to apply knowledge. The fourth level is analysis (of elements, relations, and organization principles of the body of knowledge). The fifth level is synthesis. This includes the ability of a student to generate a unique communication, produce a plan or set of operations, or derive a set of abstract relations. The sixth and highest level in Bloom’s Taxonomy is evaluation. This involves the student’s ability to judge in terms of internal evidence and in terms of external evidence in the situation. For your reference, Appendix 1 lists the learning objectives of the taxonomy and relates them to various types of learning objectives for the student. Bloom offers a similar hierarchical scheme for the affective domain. Briefly, level 1 is receiving (or attending to) the material presented. It includes awareness, willingness to receive information, and controlled or selective attention. Level 2 is responding to the material. It included acquiescence in responding, willingness to respond and satisfaction in response. Level 3 entails valuing. It encompasses acceptance, preference for, and commitment to material learned/values. Level 4 is organization (of values) into a system. Level 5, the highest level in the affective domain, involves acceptance of a generalized set of values and characterization of the world in an internally consistent manner. Appendix 2 summarizes levels in the affective domain. Within the psychomotor domain, Bloom offers no taxonomic levels. There are several possible advantages associated with the use of Blooms Taxonomy for ABSEL researchers. First, it may help researchers conceptually clarify the domain and level of learning goals that the researcher is investigating. Creators and users of simulations/exercises have cited many goals. Many of these goals are disjointed, unclear, or so all encompassing that they could not possibly be achieved by one simulation exercise. Taxonomies can help clarify our thinking with regard to what already exists and what we expect after the simulation or experiential exercise is carried out. Second a taxonomy can help us see how close--or apart- -we are in establishing, measuring, and evaluating education objectives (e.g., might we be measuring the same learning outcomes and calling them by different name?). Third, if ABSEL research is to be cogent, credible, and coherent, a standardized way to define, measure, and evaluate the learning benefits of simulations/experiential exercises must be established. Blooms Taxonomy represents a good starting point. “...through reference to the taxonomy as a set of standard classifications, teachers should be able to define such nebulous terms as those given above. This should facilitate the exchange of information about their curricular developments and evaluation devices. Such interchanges are frequently disappointing now because all too frequently what appears to be common ground between schools disappears on closer examination of the descriptive term being used.’ (Bloom, 1956, 1) A FIRST CUT AT CLASSIFYING ABSEL RESEARCH ON LEARNING AND RESEARCH DESIGNS Up to this point, we have introduced in a somewhat general fashion a taxonomy of learning. In this section of the paper, we wish to combine this taxonomy with an examination of research design, to develop a useful framework from which to view work undertaken by ABSEL researchers. Such a framework will be useful in illuminating directions for future research. As we discussed previously, Bloom’s Taxonomy envisions three qualitatively different learning outcomes: cognitive (or Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 88 knowing); affective (or feeling); and psychomotor (or doing) . Our first step was to examine eight years (1974-1977 and 1981-1984) of ABSEL research studies of learning and classify them into these categories. The results of this effort are displayed in Table 1. As can be seen, a number of interpretations and insights may be spawned from Table 1. However, it is our intention to simply note the major trends, rather than to speculate as to why such trends exist. What we consider to be the most pronounced trends are as follows: -- There are no studies directed toward the Psychomotor Domain. -- Overall, the ‘Other category was most frequently observed. As noted elsewhere, this category included all studies which do not address learning in either normative or empirical mariner. It was observed in 45% of all articles. -- The Cognitive Domain was the most frequently studied of the three learning domains, approximately 21% of all articles. Although it was not the primary purpose of this investigation to study sub-categories within the various domains, we found 80% of studies of the Cognitive Domain were at level 3 (application) and level 4 (analysis) -- A comparison of the first four years of ABSEL studies with the last four years shows a remarkably consistent pattern for all the categories. One noteworthy exception to this is the Affective Domain, where the percentage of studies has doubled from the first four years to the last four years. Our second task was to categorize the same eight years of ABSEL studies in term of research methodology. While it would be ideal to classify these same studies into one of the three categories of research designs (pre- experimental, quasi-experimental, and experimental), in practice this proved difficult. The reason it is difficult is that the categories overlap somewhat as depicted in Figure 1. We determined to evaluate the studies on their use of randomization, control groups, and experimenter control of the treatment variable. These three characteristics were selected based on a review of the research design literature, as indicators of the strength of research designs. We would not argue that these are the only possible indicators, nor that there may be some other method that is preferable to this approach. What we were looking for was an approach that would provide us with a least a crude measure of the attention ABSEL researchers have paid to research design issues. It seems reasonable to assert that studies that lack all three characteristics (randomization of subjects, control groups, and experimenter control of the treatment variable) represent pre-science studies~ Studies that have one or two of the other characteristics are weaker than studies characterized by all three factors, in terms of “goods’ science. The results of our examination of research methods is presented in the body of Table 2. Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 89 While the results of Table 2 seem self-evident, several points are worth stressing: -- The Prescriptive and ‘Other’ studies were the most frequent, ranging from a low of 61% in 1982 to a high of 82% in 1983. Generally, there is a relatively stable pattern in terms of the design categories. One notable exception to this is a decline in the Prescriptive type articles, from 42% to 31% and an increase in the ~Other category from 29% to 40% over the period studied. -- Of all the design categories listed, the ‘Control’ category consistently occurred most frequently. After “control, studies with at least two design elements ranked next in terms of the number of actual articles, while only 10% of the studies had all three design elements. Clearly, studies in this latter category can be described as having the strongest experimental design. -- While this result is not indicated in the table, we found a high cross-classification (roughly 65%) between studies in the ‘Prescriptive’ category and studies which addressed one or more of the learning domains. For example, a study which described a simulation and suggested various learning benefits from using that simulation would fall into this cross-classification. These findings suggest that much remains to be done in terms of strengthening the typical research project. DISCUSSION AND CONCLUSIONS The purpose of this paper is to examine ABSEL research in terms of a taxonomy of learning objectives and, likewise, to examine the degree to which ABSEL articles have employed fundamental research methodologies. The authors hypothesized that ABSEL research generally falls short of specifying clear learning objectives for simulations/experiential exercises and that ABSEL research generally failed to employ basic research methodologies. Furthermore, it became evident from examining the results of this analysis (Table 1) that research efforts of ABSEL members have been focused primarily or~ cognitive learning and have tended to ignore two other areas of learning~ affective, and psychomotor. In addition, within the cognitive learning domain, ABSEL researchers have tended to focus on only two levels--applications and analysis--of the six primary levels in the cognitive domain hierarchy. Thus, in terms of research on learning, it may be that one direction researchers interested in experiential learning should take is to examine the affective and psychomotor domains as well as the cognitive learning domain. In addition, even within the cognitive domain, we might profitably broaden our studies to look for evidence of learning outcomes at additional levels in the Bloomian hierarchy of cognitive learning outcomes. In particular, level 1 (Knowledge); level 2 (Intellectual Ability/Skill); level 5 (Synthesis); and level 6 (Evaluation) appear to be almost virgin territory for research. Most imperative, however, is that ABSEL researchers, whether describing or analyzing a simulation/exercise, must clearly understand and specify the learning objectives studied in terms of a standard taxonomy. Only by relating learning objectives and outcomes in a standard format (Bloom’s being an example), can ABSEL hope to establish credible and unified evidence for the benefits of exercises. In terms of research design issues, it is clear from both our review of the designs used in articles published in the annual proceedings, and from our knowledge of how we attempt to conduct research, that it is difficult to develop strong designs given the attempt to integrate research efforts with the teaching and learning context. In particular, as a professional society, we have underutilized the techniques of randomization (to achieve equivalent groups) , control groups (to rule out the most likely of alternative explanations) , and control of the treatment variable. Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 90 The problem with knowledge developed using weak designs is that it is highly suspect; and, therefore, does not provide a solid base on which to build. It would seem that an implication of our findings in this survey is that as a professional society, we need to encourage our membership to employ stronger designs and, thus, develop the reliable knowledge base essential to effective utilization of experiential exercises. Such encouragement might take place through conducting workshops at the annual meetings on design of research, establishing a recognition program for excellent research by our members, and other actions to foster attention to this critical issue. In the long run, this society will prosper as its membership is able to demonstrate they are more effective as a result of experiences and knowledge developed in association with ABSEL. A task essential to the long-term survival of this association is to strengthen our approaches to knowledge building. This will require changes in the emphasis and activities of our association. It will require us, as members, to open up in the traditional ABSEL spirit and share our questions, problems, and approaches to doing good knowledge building. Given the unique spirit of ABSEL, we can all expect to benefit from this challenge. REFERENCES Bloom, Benjamin S. (Ed) , Taxonomy of Education Objectives: The Classification of Education Goals, Handbook I: Cognitive Domain, New York, NY: David McKay Company, Inc., 1956. Brenenstuhl, Daniel C. and Ralph F. Catalanello, “An Analysis of the Impact Upon the Learning Effectiveness of Traditional Instruction, Simulation Gaming and Experiential Learning Teaching Methodologies: An Experimental Design,” Proceedings, ABSEL, Knoxville, Tennessee, 1976, pp. 463-473. Burns, Alvin C. and James W. Gentry (1977) “Some Thoughts on a Theory of the Use of Games and Experiential Exercises,” Proceedings, ABSEL, Dallas, Texas, 1977, pp. 17-20. Campbell, Donald T. and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research, Chicago, Ill. Rand McNally & Company, 1963. Gentry, James W. and Alvin C. Burns, “Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises,” Proceedings, ABSEL, Orlando, Florida, 1981, pp. 48-50. Keys, Bernard, ~A review of Learning Research in Business Gaming,” Proceedings, ABSEL, Knoxville, Tennessee, April 11-13, 1976, pp. 173-184. Krathwohl, David R., Benjamin S. Bloom, and Bertram 8. Masia, Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook II: Affective Domain, New York, NY: David McKay Company, Inc., 1956. Smith, Jerald R., “Business Game Design: From Theory to Practice,” Proceedings, ABSEL, Orlando, Florida, 1981, pp. 90-93. Wolfe, Joseph, “Comments on the Perception, Identification, and Measurement of Learning from Simulation Games,* Proceedings, ABSEL, Knoxville, Tennessee, 1976, pp. 288- 294. Wolfe, Joseph, “Research on the Learning Effectiveness of Business Simulation Games-A Review of the State of the Science,” Proceedings, ABSEL, Orlando, Florida, 1981, p. 72. Table of Contents Volume 12, 1985 One Step Beyond Naiveté: Laboratory Simulation of Strategic Management Predicting Performance Over the Course of the Simulation Congruency of Critical Performance Factors in Simulated and Real World Environments: An Exploratory Study Stimulating Learning and Creating Course Involvement Through the Use of Debates A Novel Approach to Writing Texts ABSEL Revisited: In-Class Student Involvement Increases Learning Bus-Sim Micro: An Introductory Business Simulation Demonstration Solutions Diversified: And Experiential Introduction to Management Information Systems Simulating Rigorous Analysis in Stimulation Gaming The Use of Decision Support Systems (DSS) and Operations Research/Management Science (OR/MS) Techniques to Enhance the Learning Experience of Students Participating in Computerized Simulation Problems Associated with the Administrative Use of Student Ratings Enthusiasm Awareness in the Experiential Classroom Participation Expectations of Students in Experiential Settings Using the Computer as a Planning Aid in an Applied Marketing Course Integrating Microcomputers in the Marketing Curriculum Through the Use of Marketing COMPUPROBS Giving Students Experience in Team Report Writing Description and Results of an Experiential Approach in the Teaching of Business Policy Using Real World Experiences as an Experiential Activity: the Page Technique A Guide to Writing Microcomputer Simulations The Development of a Microcomputer Distributed Processing Business Game A Monte-Carlo Approach to Interactive Gaming Single Case Experimental Design as an Alternative to the Conventional Approach Learning Theory and Research Design: How Has ABSEL Fared? Criticisms of the Use of Simulations in Economics: A Rebuttal Using the General Management Game in the Business Policy Course General Management Policy Simulations: Which Games are Popular, and Why do Professors Stop Using Games? Business Policy Content and Teaching Methods: Comparisons of the Hegarty and Summers and Boyd Findings Let's Talk to the Experiential Authors A Computerized Logistics Game for Micros Effects of Contextual Factors on Bargaining Decisions of Experiential Group Representatives The Linking of a Collective Bargaining Simulation and a Business Simulation Game - A Case Study Students are Surrogates in Arbitration Studies AIRWAYS: A Microcomputer Simulation of a Service Industry A Proposed Interactive Inventory Control Simulation Microcomputer Demonstration Exercises for a Course on Creativity and Problem Solving Processing Skills for Teachers, Trainers and Managers Developing the Competencies of Resistance to Stress and Accurate Self-Assessment Raising Elizabeth: Socializing Girls to Aspire to Careers as Managers, Secretaries or Corporate Wives Factors Influencing Perception: Gaining Understanding Through Experience Painless Computer Simulation The Effects of Differential Grade Weights on Business Game Learning levels An Investigation of the Validity of a Recommendation for Experiential Exercise Debriefing Simulation and Experiential Practices of Faculty: A Data-Based Workshop The Live Case Study: Filling the Gap Between the Case Study and the Experiential Exercise A Reactive Microcase Pedagogy Advantages of a Multigame Simulation Course Using the Logistic Function to Score Computer-Supported Games A Simulation Model to Calculate Session Time for Running Simulation Models in a Shared Resource Environment On the Core Size of Small Task Groups and the Cause for the Core Size to Emerge