ASSESSING THE EFFECTS OF A COMPUTERIZED STUDY GUIDE IN MACROECONOMIC PRINCIPLES: A STATISTICAL ANALYSIS Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 60 ASSESSING THE EFFECTS OF A COMPUTERIZED STUDY GUIDE IN MACROECONOMIC PRINCIPLES: A STATISTICAL ANALYSIS Paul L. Altieri and A. Papathanasis Central Connecticut State University1 1 We wish to thank Ronald Daigle, James Loughlin, Ki-Tai Pae and two anonymous referees for their helpful comments. We also thank the Joint Council on Economic Education for their support. ABSTRACT The purpose of this study was to test the effectiveness of a computerized study guide in reinforcing the macroeconomic concepts taught in the classroom. The paper discusses the design of the software, its testing in the field, and a statistical analysis of its effects on learning and retention. The results suggest that when gender and GPA are taken into account, this form of CAI significantly improves the comprehension of complex economic concepts. INTRODUCTION Computer Aided Instruction (CAI) has been employed in economics with most of the field work of the computer simulation and game varieties. However, the effects on learning and retention of an interactive-type study guide that allows students, especially those in large undergraduate classes, to progress at their own pace had not been adequately tested. As a result, this paper describes the design of a computerized study guide for Principles of Macroeconomics, its testing in the field, and a statistical analysis of its effects on learning and retention. LITERATURE OVERVIEW Studies of the effectiveness of CAI and simulation games, including the non-computerized type, on cognitive behavior have produced mixed results in the past. Wentworth and Lewis [16, p. 117] concluded that students exposed to a non-computerized version of an economics simulation game performed better when tested with an instrument designed to measure comprehension of economic concepts. The instrument is called TUCE (Test of Understanding in College Economics) and is published by the Joint Council on Economic Education [10]. It consists of a 30 question multiple-choice test, available in a variety of versions. Chizmar, Hiebert, and McCanney [4, p. 43], on the other hand, used CAI as an optional supplement. The test instrument did not uncover differences in scores between students who used the simulation and those who did not when the scores were adjusted for differences in GPA. A study made by Cox [5, p. 30] using eight simulations differed from other studies because the values of the variables were chosen in advance by the instructor and not by the students. Again, no significant differences in scores were detected. On the other hand, Soper cites improvements in scores were detected in the study by Booms and Kaltreider as well as in the study by Bonello, Davisson, and Jameson [11, p. 9]. Additional affirmative empirical evidence of the effectiveness of CAI was offered by the studies of (1) Paden, Dalgaard and Barr [91, (2) Swartz, Bonello, and Davisson [13], and (3) Marlin and Niss(7). The conflicting results emerging from these studies could be attributed to methodological shortcomings. First and foremost, the methods were irreproducible. Second, a number of the studies used inappropriate statistical techniques. Lastly, the studies did not include a number of variables [2). Our study avoided some of the aforementioned shortcomings by the use of the methodology given below. HYPOTHESIS We hypothesized that students in the experimental group would show a greater improvement in scores on the posttest, given at the end of the semester, over the pretest, administered at the beginning of trte semester, then those students in the control group. However, the abilities of the individual students, and whether or not they had studied economics in high school would probably also influence the improvement in scores. In addition, we agreed with Wehrs that gender might be another important factor [15, p. 1081 The students’ SAT scores and Grade Point Averages (GPA) were selected as proxies for ability. We believed that the higher the SAT and GPA, the greater improvement the student would demonstrate. METHODOLOGY According to William E. Becker [1, p. 13] Miller [8, p. 51], and Spector and Mazzeo [12, p. 37] among others, a study should provide for 1) control, b) replication, c) generalization, and d) a favorable cost-benefit ratio. Our study satisfied the control requirement by dividing the sample into two groups, an experimental group and a control group. The replication requirement was also satisfied because the results of the study could be repeated under similar circumstances and conditions by other researchers. The results could be used to make generalized inferences about the population and, thus, satisfied the generalization requirement. Finally, the relatively low cost of the project satisfied the Cost-Benefit requirement. We assumed in our model that students produced an output, understanding of economic concepts, using a vector of inputs specified by the following equation: Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 61 The variable DIF is used here as a proxy for learning. We expected that the coefficients of the variables CAI, SAT and CPA would be positive. The coefficient of PRE, on the other hand, should be negative. No a priori judgements were made concerning the sign of the SEX coefficient. DATA COLLECTION The data were collected in five sections of Principles of Macroeconomics during the spring semester of 1985. The two instructors who taught the five classes used the same textbook, study guide, course outline, testing method and spent the same amount of time on each topic. The only variation was due to the personalities of the instructors. Two of the sections served as the experimental group; the control group contained three sections. The total sample contained 147 students. After adjustment for missing data, 127 observations were left. All students in the sample were given TUCE, Macro Form A, both as a pretest and as a posttest. During the semester all students were expected to utilize the printed study guide which accompanied the textbook. Students in the experimental group, however, were allowed to substitute the use of the computerized study guide for the printed study guide. To provide the necessary incentives to perform well on the TUCE, students were allowed to substitute their performance on the TUCE (either the pretest or the posttest) for one of the regular course examinations. Finally, given that the students faced a fixed time constraint, we assumed that if they spent more time with the computer they would necessarily spend less time using the printed study guide. SOFTWARE DESCRIPTION The computer program for this study was written in BASIC for the DEC Rainbow personal computer by the authors and contained the following features. Upon signing on, the student entered his or her name and a password. Next, the main menu of topics corresponding to chapters in the textbook appeared on the screen. After selecting a particular topic, the student could choose either to read a tutorial or to proceed with a multiple choice drill session on that subject. The tutorial was designed to complement the textbook and presented the material from a different perspective. Animated graphs and overlays distinguished the tutorial from the static presentation of the standard text. For instance, in the dynamic supply and demand analysis, an animated shift of the supply curve visually demonstrates the process of reaching the new equilibrium price and quantity. In the analysis of national income, the students can watch the aggregate demand curve shift upward as government spending increases. This is not possible in the static presentation of a textbook. The program allowed the student to move back or forth through the pages of the tutorial at his or her own pace. The student could take a break, return later, and resume at the point where he or she left off. Multiple readings of the tutorial were also possible. The quiz section presented multiple-choice questions and supporting animated graphs and tables. If the student answered correctly on the first try, the message “CORRECT” flashed on the screen. Two incorrect responses in a row caused the correct answer to appear on the screen. Upon completion of the quiz, the student was presented with his score. The length of the session in minutes was automatically recorded, along with the student’s name, in a separate file accessible only to the instructor. An unanticipated problem arose when some students turned off the computer instead of exiting the program through software control. As a result, the time spent on the computer was not recorded, and thus, could not be used as an explanatory variable. RESULTS A backward-elimination procedure using the stepwise regression program of the SAS statistical package indicated that the variable HSECON did not contribute to the explanatory power of the model and could be eliminated. The data in Table 1 indicate that the coefficients of the remaining variables are statistically significant. The coefficient of the variable PRE is negative. This result was anticipated since those who did well on the pretest had less room for improvement on the post-test. The coefficients of SAT and CPA are positive, as expected. The coefficient of CAl suggests that students who used the computerized study guide show a six percentage point improvement in mean scores compared to other students. Similarly the coefficients of CPA and SEX suggest a five and seven percentage point differential respectively. The coefficient of SEX being negative suggests that males may be more comfortable with the format of TUCE than females. Although the R2 and the adjusted R2 of the model are not high, the computed F-statistic allows one to reject the null hypothesis that all coefficients are simultaneously zero. Finally, the correlation matrix of Table 2, together with the Durbin-Watson statistic and analysis of the residuals, suggest that the model does not suffer from multicollinearitv and that it is correctly specified. CONCLUSIONS The purpose of this study was to test the effectiveness of a computerized study guide in reinforcing the macroeconomic concepts taught in the classroom. We felt that the interactive approach of the computer assisted instruction would be more effective than the Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 62 traditional study guide and simulation game variety of CAI. The results suggest that when gender and CPA are taken into account, the computerized study guide form of CAI significantly improves the comprehension of economic concepts. REFERENCES [1] Becker, William E., “Economic Education Research: Part I, Issues and Questions,” Journal of Economic Education, Vol. 14, No. 1, (Winter 1983), pp. 10-18. [2] __________________, “Economic Education Research: Part II, New Directions in Theoretical Model Building,” Journal of Economic Education, Vol. 14, No. 2, (Spring 1983), pp. 4-10. [3] __________________, “Economic Education Research: Part III, Statistical Estimation Methods, Journal of Economic Education, Vol. 14, No. 3, (Summer 1983), pp. 4-15. [4] Chizinar, J.F., Hiebert, L.D., and Barnard J. McCanney, “Assessing the impact of an instructional innovation on achievement differentials: The case of CAI,” Journal of Economic Education, (Fall 1977), Vol. 9, No. 1, pp. 42-46. [5] Cox, Steven R., “Computer Assisted Instruction and student performance in macroeconomic principles,” Journal of Economic Education, Vol. 6, No. 1, (Fall 1974), pp. 29-37. [6] Hu, Teh Wei, Econometrics, University Park Press, Baltimore, 1973. 17] Marlin and Niss, “The advanced learning system,” Journal of Economic Education, Vol. 13, No. 2, (Summer 1982), pp. 26-39. [8] Miller, Jimmie C., “Technical Efficiency in the Production of Economic Knowledge,” Journal of Economic Education, Vol. 13, No. 2, (Summer 1982), pp. 3-13. [9] Paden, Dalgaard and Barr, “A decade of CAI,” Journal of Economic Education, Vol. 9, No. 1, (Fall 1977), pp. 14-20. [10] Saunders, Phillip, Revised Test of Understanding in College Economics, Interpretive Manual, Joint Council on Economic Education, 1981. [11] Soper, Johm, “CAI in Economics: A survey,” Journal of Economic Education, Vol. 6, No. 1, (Fall 1974), pp. 1-28. [12] Spector Lee C. and Michael Mazzeo, “Probit Analysis and Economic Education,” Journal of Economic Education, Vol. 11, No. 1, (Spring 1980), pp. 37-44. [13] Swartz, T.R., Frank J. Bonello and William I. Davisson, “The misuse of TUCE,” Journal of Economic Education, Vol. 12, No. 1, (Winter 1980), pp. 23-33. [14] Theil, Henri, Principles of Econometrics, John Wiley, 1971. [15] Wehrs, William, “Incentive Structure and the TUCE,” Journal of Economic Education, Vol. 9, No. 2, (Spring 1978), pp. 107-110. [16] Wentworth, R.D., and Darrel Lewis, “An Evaluation of the use of the Marketplace game in Junior College Economics,” Journal of Economic Education, Vol. 6, No. 2, (Spring 1975), pp. 113-119. Table of Contents Volume 13, 1986 How to Use Microcomputers in Human Resource Management Courses What's My Line? An Exercise in Job Analysis, Description and Classification Relationships Between Team Cohesion Dimensions and Business Game Performance Four Factors Affecting Group Performance in Business Policy Simulations Enhancing Strategic Management Goal-Setting Skills in the Business Policy Course Using a Business Simulation in the Principles of Management Course - Learning Outcomes and Perceptions Personal Computers: Drexel's Experience Using Psychological Type to Enhance Negotiation Skills The Effects of Conflict Handling Styles on Negotiation Behavior: An Analysis of an Experiential Exercise An Experiential Test of Bargaining Decisions and Representation Self-Grading of Examination by Students, A Viable Alternative to Traditional Procedures Perceived Instructor Enthusiasm and Student Achievement An Experiential Approach to Teaching Subordinate-Oriented Communication Assessing the Effects of a Computerized Study Guide in Macroeconomic Principles: A Statistical Analysis A Sales Management Simulation for the PC an Integrative Tool for the Sales Management Course Teaching Re-evaluation of Salespersons Through the Use of a Simulation Game A Test of Student Performance and Attitudes and Performance Under Varying Game Conditions Developing and Testing Airways: A Marketing Simulation Experiential Learning Revisited: Some Thoughts on Designing More Adaptive Management Education Programs Developing the Competencies of Creativity and Accurate Self-Assessment Self-View and Norms: Some Longitudinal Laboratory Results Strategic Planning with an Experiential Case The ABSEL Research Consortium: Preliminary Steps and Position Papers An Analysis of ABSEL Conference papers (1974-1985) The Creation and Operation of a Data-base System for ABSEL Research Funding for a Research Consortium on Simulation and Experiential Learning Simulation - Indoctrination or Learning The Dilemma In Evaluating Classroom Innovations Personality Variables on Group Cohesion, Team Participation and Total Learning Justifications for and Problems in Developing and Using Computerized Experiential Activities Management Competency Models and the Life-Long Learning Project: What Role for ABSEL External Validity of Business Games Restructuring the University - An Experiential Exercise The Surrogate Class Exercise Images of Effectiveness: A Classroom Exercise The Algebran Industry: A Tutorial Framework Another Chance: A Multi-Leveled, Macro Business Game for Micro-Computers A Water Quality Management Simulation Game Acquisitions? Divestitures? Progress Report on a Conglomerate Game Enterprise: A Multi-Purpose Management Development Simulation Marketer: A Microcomputer Simulation in a High Tech Industry A Pedagogical Analysis of Alternative Tax Elections for Exploration Costs in the Mining Industry Using of Multiple Microcomputers Application Programs to Teach Fundamental Business Concepts and Practices Demonstrating the Role of Simulation in Strategic Operations Planning: A Case Study in Bank Check Processing Location Analysis The Subjective Side of the Decision Support System: A Pitfall for the Panacea A Group-Based Procedure for Revealing Judgmental Heuristics and Biases An Experiential Exercise in Bayesian Decision-Making Values for Selected Parameters in Physical Distribution Simulation and Games Computerized Business Management Simulations for Tyros The Management Decision Laboratory at New York University An Example: The Use of Management Games on Microcomputers by Computer Novices Enhancing Mainframe Simulation Via Microcomputers: Designing Decision Support Systems Integrating Personal Computers Into a Course as a Decision Support Tool A Decision Support System for Capital Funds Forecasting Performance and Attitudinal Affects of a Decision Support Package in a Business Game Using the Real World: Interviews with Woman Mangers The Cooperatively-Authored Student Handbook International Buying: An Experiential Exercise The Research/Teaching Interface: Turning a Pretest into an Experiential Exercise Building Microcomputer Business Simulations Simulation with Discrete and Continuous Mathematical Modeling An Interpolation Approach to Developing mathematical Functions for Business Simulations The Design of Simulation Models Using Thought Organizers The Assessment Center as a Teaching/Learning Device An Approach to Meeting the AACSB Guidelines for Introducing Skills and Personal Characteristics (SACs) Using an Interclass Simulating and Role-Playing Pedagogy Experiential Learning about the World of Work: A Program for Primary and Secondary School Educators Assessment of Relevant Personal Characteristics of Potential Entrepreneurs