SINGLE CASE EXPERIMENTAL DESIGN AS AN ALTERNATIVE TO THE CONVENTIONAL APPROACH Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 82 SINGLE CASE EXPERIMENTAL DESIGN AS AN ALTERNATIVE TO THE CONVENTIONAL APPROACH Alvin C. Burns, Louisiana State University Jeanne H. Burns, Louisiana State University ABSTRACT The authors note the relative infrequency of rigorous experimental design studies in our literature. In an explanation of why, five troublesome issues are described: (1) ethical concerns, (2) generalizability problems, (3) administration difficulties, (4) use of average analyses, and (5) statistical significance versus meaningful change. From here, the individual nature of learning is reviewed and the subsequent need for an individually-based assessment technique claimed. Single Case Experimental Design (SCED), a method widely adopted by clinical psychologists, is described. It is pointed out how this approach overcomes the aforementioned Conventional Approach problems. Considerations associated with SCED are discussed, and a call is made for business education researchers to try SCED. INTRODUCTION Numerous writings and research endeavors have been aimed at improving our ability to assess the consequences of various pedagogies on learning of business discipline content. Perusal of the ABSEL and other business education- related literature suggests to us that one might classify these investigations across a continuum of experimental design rigor. Figure 1 offers such a continuum superimposed with a hypothetical curve relating the relative incidence of each type of study in our literature. An informal analysis of the 1984 ABSEL Proceedings (4] lends credence to this claim. When one deletes papers not dealing specifically with the evaluation of education effects derived from gaming experiences, the results are as follows: a plurality is anecdotal, or personal observations of the efficacy of a specific approach; the next most coon type is the “Single Group Ex Post?’ study in which the performance of a group of students is evaluated after the fact and reported without a pretest measure; less frequent are studies where an experimental group’s performance is compared to a pretest or some control group’s performance. Finally, only a small fraction is of the “Group-To-Group-To-Control” category wherein two or more alternative pedagogies are compared to each other as well as against a baseline or control group’s performance. If the 1984 Proceedings are representative, the vast majority of our research efforts is low in rigor. One need not read far in the experimental design methodology area to find damning criticism of the first two types of studies and cautionary statements on the third one. It is probable that the low incidence of rigorous experimental designs is caused by their requirements of strict control procedures which deter researchers, Consequently, the purpose of this paper is to broach Single Case Experimental Design (SCED) as a technique for assessing learning effects of various gaming alternatives. The notion of an N=1 methodology will undoubtedly be uncomfortable to most readers, but it is a viable approach. Furthermore, it is well- suited to the education process, or so will be argued here. This paper will proceed with a review of the conventional experimental design approach, leading into a discussion of the difficulties and issues associated with its applications. The paper will next address some basic assumptions about the nature of business education and student learning in a gaming environment to illustrate how SCED is an appropriate technique. The remainder of the paper describes SCED and closes with some practical considerations one should address in using SCED. THE CONVENTIONAL APPROACH WITH HIGH EXPERIMENTAL DESIGN RIGOR Business education researchers are taught that Group-To- Group-To-Control designs are the most efficient ones available in our search for learning effects differences across pedagogies. For example, the study by Burns and Sherrell [21 utilized a Control Group and three Treatment Groups to compare the effects of microcomputer simulation, experiential exercise, and case study on the learning of marketing subject matter. Although this study relied on path analysis, an earlier paper [1] reported no significant differences in the learning of content (as measured by performance on an objective test), but more positive attitudes were derived from the microcomputer simulation treatment than from other treatments. Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 83 The findings of this study are irrelevant to the central point of this paper, however. What is relevant concerns the practical difficulties and more esoteric issues of conventional experimental design. No less than five separate ones are identifiable. Each will be described briefly. 1. Ethical Concerns. Because experimental design guarantees cause and effect results, there are questions so to the ethics of administering (or withholding) treatments which result in lower performance levels. Assuming that higher performance is desired, the researcher is left with the problem of resolving the discrepancy between ethical concerns and his/her desire to report significant findings. Burns and Sherrell did not even debrief the students in their study. 2. Generalizability. Questions of internal and external validity invariably arise in the application of experimental designs. Elaborate control procedures must be employed to eliminate the intrusion of alternative hypotheses. Also, the artificiality of the experiment delimits its generalizability. Burns and Sherrell admit that the attitude questionnaire was administered before students could compare experiences: students were not told about the objective test administered in the next class; the variation of the experiment from regular class activities may have heightened attitude changes. 3. Practical Administration Difficulties. Anyone who has attempted rigorous research in the social sciences becomes intimately aware of the practical difficulties involved. Groups must be equivalent; outside influences must be controlled; demand effects must be minimized; drop-outs must be anticipated; sampling must be uniform; treatments must be verified; and so on. Burns and Sherrell relied on a judging by MBA students for subject matter comparability across treatments; four separate instructors were needed in each session; random splits of the class were necessary; students in different treatments were not allowed to interact during the experiment, etc. 4. Use of Average Analyses. Analyses of variance and t- tests used to determine significant differences between experimental groups rely on the mean. While statistically elegant and unquestioned from a mathematical standpoint, this central tendency measure Ignores individual variability. For instance, studies have reported differences between males and females found in post hoc analyses. See [31 for example. Burns and Sherrell did not address sex differences, but past research findings suggest that they would have found them. If such gross individual differences exist, there is sufficient evidence to be concerned with the operation of more subtle individual differences. There are probably interactions between individual differences and treatments, as well. 5. Statistical Significance Versus Meaningful Change. The Conventional Approach is inherently conservative in stating results. Low alpha levels (e.g., .01, .05, or even .LO) are used to assess the significance of differences between group means. Burns and Sherrell adopted the .10 level. Despite convention, the issue of practical significance versus statistical significance stands unresolved. In fact, any differences are taken by practitioners as meaningful. In Summary, there are a number of bothersome controversies which detract from rigorous conventional experimental designs. SCED, on the other hand, does not Buffer from any of these problems, as will be noted later. As a prelude to our description of SCED, we believe it i~ worthwhile to re-examine what we are attempting to accomplish as business discipline educators in the first place. PREMISES ON THE NATURE OF EDUCATION AND LEARN INC IN A CANING ENVIRONMENT Experimental design is rightly viewed as a family of techniques and procedures applicable to an immense variety of problem areas. Before adopting any part of this family of tools, it seems advantageous to review the attributes of the phenomenon under investigation. We have formulated five premises regarding learning and learning performance evaluation. In our view, these premises are intuitively tautological, although some readers may debate the finer points. Also, we must admit that our premises are not original even though we have not provided extensive documentation. 1. Learning is a psychological construct. Researchers define learning variously as some relatively permanent change in behavior due to an experience. More important, the notion of learning is derived from psychology, the science of individual behavior. Accordingly, learning occurs on a personal basis and must be assessed as a change from one level of operation to another as a consequence of some event experienced between the two. 2. Learning is subject to individual differences. Individuals respond differently to different learning situations. Evidence of individual differences in learning has already been cited, but it is important to reiterate that these findings are typically discovered when researchers sift through data bases a theoretically. It is more appropriate to include individual differences a priori, expecting them to operate, and accounting for them in performance results. 3. Learning occurs without necessity of statistical significance. Learning is independent of statistical significance as any documented change in behavior or mental state qualifies is learning by definition. Statistical significance, as any researcher knows, addresses sampling error alone and may be manipulated by inflating the sample size. An N=1 methodology obviates sample size entering into the evaluation equation. 4. Learning should be measured longitudinally. Certainly in education, there is the goal of persistence of change over time. Whatever the topic, an educator surely wants learning to be retained for the long term, and, ideally, for a lifetime. In sharp contrast are one-shot measures of learning such as objective tests which are commonly used. The temporal aspect to behavioral change must be addressed. 5. Education is inherently an intervention process. If one looks at education from the (typical) student’s side, the unfortunate truth is that it is an imposition on his/her freedom. With few exceptions, students view learning as distasteful and seek to avoid it. At the least, they attempt to minimize their exposure to it. The educator, then, can best view his/her role as a change agent who intervenes the student’s aversion tendencies. Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 84 SINGLE CASE EXPERIMENTAL DESIGN DESCRIBED The scenario established thus far has portrayed rigorous conventional group-to-group experimental design as fraught with unresolved problems. It has also laid the groundwork for the claim that an individually-based evaluation procedure should be used. As luck would have it, another discipline encountered the identical problems and derived the same conclusions some years ago. That discipline was clinical psychology, and from its special needs sprang the single case experimental design methodology about to be described. Admittedly, clinical psychology addresses an entirely different dimension of human behavior - abnormal behavior. But it is parallel to education in that clinical psychologists attempt to change an individual’s behavior from one level to another through intervention. Regardless of the circumstances of administration, as practitioners, clinical psychologists must deal with individuals. Moreover, it is impractical for them to form group experiments except in rare instances. Consequently, SCED is relied upon heavily. Figure 2 illustrates the SCED approach. It begins with a series of Baseline Observations which show some consistency even though variability is present. Once a stable baseline recording has been attained, an Intervention Phase or treatment is administered. The Results Observations are taken and a comparison made between the Baseline and the Results measures. In the illustration, there has been a stable improvement in the Performance Level. For now, it will be claimed that the improvement was caused by the Intervention. As can be seen, the SCED approach is extremely simple in concept: although, some questions are apparent at the onset. Before addressing them, the advantages of SCED over the Conventional Approach should be pointed out. First, SCED focuses on variability in the individual. It ascribes all analysis to the individual level and is thus consistent with our premises on learning. Second, SCED makes use of repeated, frequent measures. For both the baseline and results stages, the performance level is monitored several times to assess reliability and stability. This aspect of SCED overcomes the one-shot measurement objections voiced earlier, and it takes into account the longevity of the change. Third, SCED allows for changing treatments within a design. That is, if the Results level failed to show improvement over the Baseline, the researcher can change treatments within the same individual In a series of “ABABAB…”experiments in the search for improvement. Finally, the SCED methodology is ideally suited to intervention-based learning assessment. A final set of advantages pertains to the five difficulties and issues discussed with respect to conventional experimental designs. In the interest of conservation of space, Figure 3 compares the Conventional Approach with SCED for each one. Inspection of the Figure reveals that SCED resolves the difficulties in all cases. SOME PRACTICAL CONSIDERATIONS WITH THE USE OF SCED It would be nice to posit SCED as a panacea to all our problems, but unfortunately, that is not the case, for there are some considerations which must be highlighted in this introduction to the technique. The first consideration pertains to statistical analysis. Because N=I, serious assumptions violations are encountered with SCED. That is, ANOVA and t-tests should not be used without adjusting the error terms to account for intercorrelations of observations. The more acceptable solution, however, is to make use of time series analysis techniques which account for changes in both level and slope from one stage to the next. Time series, one will recall, also explicitly assumes observations of the same population over time. Thus, for those who desire statistical analysis, a technique is available; however, it should be noted that controversy exists in clinical psychology as to whether or not statistical analysis is appropriate or necessary (see Hersen and Barlow [5] for elaboration.) A separate consideration concerns the desire for pedagogical generalizability. Beyond the learning effects on an individual student level, there is a need to evaluate and communicate regularities. To this end, SCED requires inductive logic. Here, one would seek uniformities across individuals, treatments, and circumstances. Probably the resolution would require a meta-analysis of a myriad of published SCED studies. A final consideration discussed here is that SCED requires persistent instructor/administrator involvement, or at the least, a system in which the learner can operate under self- monitoring. With the conventional approach, a few groups of students could be administered separate treatments, thus conserving the administrator’s time and energy. With SCED, however, there is much greater demand on the administrator to effect individual pedagogies. Since the educator is not a clinical psychologist, and students are not Developments in Business Simulation & Experiential Exercises, Volume 12, 1985 85 patients with disorders, some system would need to be developed. Learning laboratories and microcomputer-driven CAT environments would appear to place us on the technological brink of permitting extensive SCED adoption. CONCLUSION The conventional experimental design approach is firmly entrenched, and this paper will hardly activate a methodological revolution. Still, the initial observation as to that. infrequent appearance of rigorous experimentation in our published research is valid. When one reviews the difficulties of adhering to the requirements of the conventional approach, one comes to realize that there is an opportunity for an alternative method to be applied. We have sought to illustrate how the Single Case Experimental Design approach is actually better suited to the analysis of individual leaning and more faithful to our assumptions about the educational process. The technique is not failsafe, nor is it without unique requirements. But it does overcome most of the difficulties associated with the conventional approach and certainly deserves trial runs. REFERENCES [1] Burns, Alvin C. and Dan L. Sherrell, “An Empirical Test of Microcomputer Simulation as an Alternative for the Teaching of a Marketing Topic,” in An Assessment of Marketing Thought and Practice. 1982 Educators’ Conference Proceedings, Bruce Walker et. al. (eds, 1982. pp. 122-126. [2] Burns. Alvin C. and Dan L. Sherrell, “A Path Analytic Study of the Effects of Alternative Pedagogies,” Developments En Business Simulation And Experiential Learning. Proceedings of the 1984 ABSEL Conference, David Currie and James Gentry, (eds.), 1984, pp. 115-119. [3] Chilsom, Thomas A., Parameswar Krishnakumar, and James P. Clay, “Are Computer Simulations Sexist?,” Experiential Learning Enters The 80’s. Proceedings of the 1930 ABSEL Conference, Daniel C. Brenenstuhl and William Biggs (eds.), 1980, pp. 160- 164. [4] Currie, David and James W. Gentry, (eds, Developments in Business Simulation and Experiential Learning, Proceedings of the 1984 ABSEL Conference, 1984. [5] Herson, Michel and David W. Barlow, Single Case Experimental Designs: Strategies For Studying Behavior Change, Pergamon Press, New York, 1976. Table of Contents Volume 12, 1985 One Step Beyond Naiveté: Laboratory Simulation of Strategic Management Predicting Performance Over the Course of the Simulation Congruency of Critical Performance Factors in Simulated and Real World Environments: An Exploratory Study Stimulating Learning and Creating Course Involvement Through the Use of Debates A Novel Approach to Writing Texts ABSEL Revisited: In-Class Student Involvement Increases Learning Bus-Sim Micro: An Introductory Business Simulation Demonstration Solutions Diversified: And Experiential Introduction to Management Information Systems Simulating Rigorous Analysis in Stimulation Gaming The Use of Decision Support Systems (DSS) and Operations Research/Management Science (OR/MS) Techniques to Enhance the Learning Experience of Students Participating in Computerized Simulation Problems Associated with the Administrative Use of Student Ratings Enthusiasm Awareness in the Experiential Classroom Participation Expectations of Students in Experiential Settings Using the Computer as a Planning Aid in an Applied Marketing Course Integrating Microcomputers in the Marketing Curriculum Through the Use of Marketing COMPUPROBS Giving Students Experience in Team Report Writing Description and Results of an Experiential Approach in the Teaching of Business Policy Using Real World Experiences as an Experiential Activity: the Page Technique A Guide to Writing Microcomputer Simulations The Development of a Microcomputer Distributed Processing Business Game A Monte-Carlo Approach to Interactive Gaming Single Case Experimental Design as an Alternative to the Conventional Approach Learning Theory and Research Design: How Has ABSEL Fared? Criticisms of the Use of Simulations in Economics: A Rebuttal Using the General Management Game in the Business Policy Course General Management Policy Simulations: Which Games are Popular, and Why do Professors Stop Using Games? Business Policy Content and Teaching Methods: Comparisons of the Hegarty and Summers and Boyd Findings Let's Talk to the Experiential Authors A Computerized Logistics Game for Micros Effects of Contextual Factors on Bargaining Decisions of Experiential Group Representatives The Linking of a Collective Bargaining Simulation and a Business Simulation Game - A Case Study Students are Surrogates in Arbitration Studies AIRWAYS: A Microcomputer Simulation of a Service Industry A Proposed Interactive Inventory Control Simulation Microcomputer Demonstration Exercises for a Course on Creativity and Problem Solving Processing Skills for Teachers, Trainers and Managers Developing the Competencies of Resistance to Stress and Accurate Self-Assessment Raising Elizabeth: Socializing Girls to Aspire to Careers as Managers, Secretaries or Corporate Wives Factors Influencing Perception: Gaining Understanding Through Experience Painless Computer Simulation The Effects of Differential Grade Weights on Business Game Learning levels An Investigation of the Validity of a Recommendation for Experiential Exercise Debriefing Simulation and Experiential Practices of Faculty: A Data-Based Workshop The Live Case Study: Filling the Gap Between the Case Study and the Experiential Exercise A Reactive Microcase Pedagogy Advantages of a Multigame Simulation Course Using the Logistic Function to Score Computer-Supported Games A Simulation Model to Calculate Session Time for Running Simulation Models in a Shared Resource Environment On the Core Size of Small Task Groups and the Cause for the Core Size to Emerge