teh Australian Journal of Educational Technology An evaluation of computer assisted learning in geography in Singapore George P. L. Teh Nanyang Technological University, Singapore Barry J. Fraser Curtin University of Technology, Perth Six computer assisted learning (CAL) modules in geography were developed by the investigators and evaluated in the unique milieu of Singapore schools in terms of impact on achievement, attitudes and classroom psychosocial environment. A sample of 671 students from the second year of high school were assigned randomly to either a CAL or control group. In contrast to past research, effect sizes were relatively large and ranged from 1.0 to 3.5 standard deviations. Compared with control students, CAL students had higher achievement and attitude scores and perceived their classes as having greater gender equity, investigation, innovation and resource adequacy. This evaluation of computer assisted learning (CAL) courseware is distinctive in two important ways. First, the research was conducted in the unique milieu of the Singapore school system. Second, in addition to investigation of the impact of CAL on traditional achievement and attitude outcomes, the present evaluation broke new ground in that it also involved the development and use of an instrument which assessed the psychosocial environment in CAL classrooms. The Singapore education system is different from that of many other countries in that it is highly centralised (with almost every school using the same prescribed textbooks), the mode of instruction is essentially expository and achievement-oriented, and schools are graded for excellence based solely on the students' academic performance. Educational applications of computers were introduced fairly extensively into Singaporean schools only in 1980, with the focus initially on computer science and low-level computer literacy courses at the secondary school 56 Australian Journal of Educational Technology, 1994, 10(1) level. Computer assisted learning (CAL) was introduced to the schools in 1986, mainly for remediation and enrichment, but no Singaporean school develops its own CAL courseware. In Singapore, there is a dearth of research on the effects of CAL. Moreover, the only three research studies that have been conducted (Ong & Lee-Leck, 1986; Low, 1988; Woo-Tan, 1989) involved small and non-random samples, a short duration of the experimental treatment, and a narrow scope of the content selected. Because of the lack of dependable research information, and in view of the potential that research conducted in the unique educational context of Singapore has for our understanding of computer assisted learning, the present study was conceptualised and conducted. The main purposes of the present study were to develop six CAL modules on the topic of decision-making in geography, and to evaluate this innovation in terms of its impact upon student achievement, student attitudes and the classroom environment. The CAL and traditional teaching approaches The CAL courseware developed for this study covered the topic of decision-making in geography and was designed specifically for a six- week segment in the Singapore school geography syllabus. This CAL courseware consisted of a curriculum-based set of instructional modules specifically developed for the target population in Singapore. It was content-specific and curriculum-specific. That is, the content of the CAL modules followed very closely the curriculum prescribed by the Singapore Ministry of Education. The length of each CAL module was determined by the length of each teaching period in the schools. Even the day and time for undertaking the CAL modules took cognisance of the rigid timetabling in the schools. The CAL approach consisted of a sequence of instructions consisting of learning activities for the concept of decision-making, followed by a prescribed set of exercises. During the CAL lessons, students were presented with the necessary information. Students also were guided with instructions on the computer screens, were provided with practice exercises, and had their learning assessed. Feedback was provided contingent upon correct and incorrect responding. In order to enhance the validity of conclusions from the evaluative study, a control group, which studied the same topic via the direct expository teaching methods common in Singapore, also was included in the study. The traditional expository method used by the control group was similar to the CAL approach in that it consisted of six modules (lasting six weeks) of two lessons per module. In terms of curriculum time, each module covered two 35-minute lessons. Thus, the CAL and control groups learned Teh and Fraser 57 the same content within the same time span. The same teacher taught the lessons for both groups. Furthermore, similar sets of exercises and worksheets were given to both groups in order to enhance internal validity. The exercises for the CAL group were stored in the computer, whereas the exercises for the control group were given out as worksheets. The traditional method consisted of a sequence of instructions with the teacher using the direct expository teaching method. The teacher initiated, directed and controlled the learning. Teaching resources such as slides, charts, diagrams and transparencies were used. Students were provided with worked examples and solutions displayed on the overhead projector. Students then were required to complete exercises in the worksheets. Any support for the students in their understanding of what they were doing came from the teacher, who had provided the initial explanation and the worksheets which served to reinforce the topics taught. Interactions within the class were generally between the teacher and the class as a whole, with limited one-to-one interaction between teacher and student, or between student and student. Method Sample The study involved 12 teachers, each in a different randomly selected school. In order to reduce the 'teacher effect', each teacher taught one experimental and one control class. All schools were coeducational. The total number of students in these 24 classes was 671 (348 in the experimental group and 323 in the control group). Approximately equal numbers of males and females made up the sample. Students were slow learners in their second year of high school in Singapore (referred to as 'Secondary Two Normal' students). Geography Achievement test (GAT) and Semantic Differential Inventory (SDI) The instruments used to measure achievement in and attitude towards geography were developed and validated for the present study. The Geography Achievement Test (GAT) is a 30-item multiple-choice test which assesses the topic of decision-making in geography. The Semantic Differential Inventory (SDI) is a 20-item semantic differential instrument which measures students' attitudes towards learning geography. The alpha reliability coefficient for the whole sample was found to be 0.95 for the 30-item GAT and 0.94 for the 20-item SDI. Geography Classroom Environment Inventory (GCEI) It is now a quarter of a century since the Learning Environment Inventory was used as part of the research and evaluation activities of Harvard 58 Australian Journal of Educational Technology, 1994, 10(1) Project Physics (Welch & Walberg, 1972), and Moos began developing social climate scales for a wide variety of human environments, including the Classroom Environment Scale for use in school settings (Moos & Trickett, 1987). Since that time, the field of classroom environment research has flourished (Fraser, 1986, 1994; Fraser & Walberg, 1991). One promising but largely neglected use of student perceptions is as a source of process criteria in evaluating educational innovations (Fraser, 1981). For example, an evaluation of Harvard Project Physics showed that student perceptions of classroom environment differentiated revealingly between curricula, even when various outcome measures showed negligible differences (Welch & Walberg, 1972). The research reported in this paper is distinctive in that it provides one of the first evaluations of CAL which investigated the psychosocial environment of CAL classrooms. Because hitherto there has existed no learning environment instrument which has been tailor- made specifically for use in classrooms using computer assisted learning, the present study filled a gap by developing and validating such an instrument. The final version of the new instrument, the Geography Classroom Environment Inventory (GCEI), has four Likert-type scales. The response alternatives for each item are 'almost never', 'seldom', 'sometimes', 'often' and 'very often'. The initial GCEI instrument contained the eight scales of teacher concern, participation, gender equity, investigation, innovation, differentiation, organisation and resource adequacy; but four scales (namely, teacher concern, participation, differentiation, and organisation) were omitted after item and factor analyses. Each scale was selected because of its relevance to the unique environment of CAL classes (Teh & Fraser, 1993). For example, gender equity was included because a decade of research in the CAL environment has shown that the use of computers maintained and exaggerated inequities (Sutton, 1991), that equity issues are complex (Schubert, 1986, Sutton, 1991), and that there are gender differences in achievement and attitudes towards computer usage (Hattie & Fitzgerald, 1987; Sutton, 1991). Although a strong tradition in CAL research at all school levels has been to investigate the effectiveness of the usage of CAL in a specific cognate area (Bangert-Drowns, Kulik & Kulik, 1985; Roblyer, Castine & King, 1988; Kulik & Kulik, 1991), past CAL research seldom has examined the potential of computer usage in facilitating and promoting student investigation in the classroom. Investigation therefore was another dimension that was incorporated into the GCEI instrument. Eight items are contained in the final version of each scale. Typical items contained in the GCEI are "The teacher pays more attention to boys' questions than to girls' questions" (gender equity), "Students carry out investigations to answer questions coming from class discussions" Teh and Fraser 59 (investigation), "New and different ways of teaching are used in this class" (innovation), and "There are enough computer programs available for our lessons" (resource adequacy). The scoring direction is reversed for almost half of the 32 items in the GCEI. In developing the GCEI, cognisance was taken of Moos's three general dimensions as they apply to all human environments (Moos, 1974). These three general dimensions are relationship dimensions (the nature and intensity of personal relationships within the environment), personal development dimensions (the basic directions along which personal growth and self-enhancement tend to occur) and system maintenance and system change dimensions (the extent to which the environment is orderly, clear in expectations, maintains control and is responsive to change). Table 1 shows that the four scales in the final version of the GCEI provide reasonable coverage of the three different basic types of dimensions proposed by Moos. Table 1. Descriptive information for GCEI scales Scale name Description Moos's classification Gender equity Extent to which boys and girls are treated equally by the teacher Relationship Investigation Extent to which the skills and processes of inquiry are used in problem-solving and investigation Personal development Innovation Extent to which the teacher plans new and varying activities and techniques, and encourages students to think creatively System maintenance Resource adequacy Extent to which the computer hardware and software are adequate System maintenance The development of the GCEI also followed the following steps. First, a comprehensive review of the literature on computer assisted learning was undertaken to identify scales considered important in this unique environment (Plomp & Pelgrum, 1991). Second, extensive interviewing of teachers and students ensured that the GCEI's initial scales and individual items were considered salient by teachers and students. Third, several computer education experts vetted scales and items and agreed that the constructs were relevant. Fourth, following extensive field testing, item and factor analyses were used to refine the original 80-item version with 8 scales to form the final 32-item version with 4 scales (Teh & Fraser, 1993). The Appendix contains a copy of the final version of the GCEI. The first, second, third and fourth item in each block in the Appendix assesses, respectively, gender equity, investigation, innovation and resource 60 Australian Journal of Educational Technology, 1994, 10(1) adequacy. The number circled represents the score allocated except for items with R in the for teacher's use column; these items are scored in the reverse manner. Omitted or invalidly answered items are scored 3. Validation statistics for the GCEI Table 2 shows the statistics obtained with the 348 students in the experimental (computer) group for each of the GCEI scale's internal consistency (alpha reliability) and discriminant validity (correlations between scales). These indices were calculated using the individual as the unit of statistical analysis. Data in Table 2 generally show that, for this sample, GCEI scales displayed adequate internal consistency reliability (with alpha coefficients ranging from 0.52 to 0.68) and discriminant validity (with mean correlations ranging from 0.01 to 0.45). Validation data were fairly similar for the control group. Table 2. Internal consistency reliability (alpha coefficient), discriminant validity (scale intercorrelations), and ANOVA results (F and [eta squared]) for class membership differences for each scale in the modified version of GCEI for the experimental group. Scale Number of items Alpha reliability Scale intercorrelations ANOVA results GE IV IN RA F Gender equity 8 0.67 0.01 0.22 0.34 17.46** 0.38 Investigation 8 0.65 0.45 0.10 49.30** 0.64 Innovation 8 0.52 0.20 42.12** 0.60 Resource adequacy 8 0.68 31.27** 0.53 ** p<0.01 Table 2 also provides information about each scale's ability to differentiate between the perceptions of students in different classrooms. These results were obtained by performing for each scale a one-way ANOVA, with class membership as the main effect and using the individual as the unit of analysis. Results of these analyses reported in Table 2 indicate that each of the four scales differentiated significantly (p<0.01) between the perceptions of students in different classrooms. The statistic, which represents the proportion of variance in environment scores accounted for by class membership, ranged from 0.38 to 0.64 for the various scales. The results for the control group were fairly similar. Teh and Fraser 61 Impact of CAL on achievement, attitudes and classroom environment A comprehensive synthesis of 134 past meta-analyses of 7,827 individual studies of factors affecting student achievement by Fraser, Walberg, Welch and Hattie (1987) has shown that the average effect size in past studies in education is only 0.4 standard deviations. For computer-based instructional programs, in particular, a review by Roblyer, Castine and King (1988) also revealed an average effect size of 0.4 standard deviations, while Kulik and Kulik's (1991) meta-analysis of computer-based learning effectiveness revealed a value of 0.3 standard deviations. To investigate differences in students' scores on the achievement, attitude and classroom environment measures between the experimental (computer) and control (non-computer) group, ANCOVA procedures were computed separately for each of the six outcome measures. Three different measures (namely, students' national primary school leaving examination scores, their semestral assessment scores and their pretest scores) were used as covariates. A summary of the results for the significance of differences between the computer and non-computer groups are reported in Table 3. The effect size (ie., the number of standard deviations of difference between the experimental CAL group and the control group) for each outcome also is reported in Table 3. Table 3. A comparison of experimental and control groups on achievement, attitude and classroom environment outcomes Scale Number of items F Effect size [1] Achievement 30 5824.59** 3.5 Attitude 20 630.13** 1.4 Classroom Environment - Gender equity - Investigation - Innovation - Resource adequacy 8 8 8 8 219.66** 809.14** 703.63** 504.28** 1.0 1.9 1.7 1.5 ** p<0.01 [1] The effect size is the difference between the means of the experimental and control groups divided by the standard deviations for the control group. 62 Australian Journal of Educational Technology, 1994, 10(1) Table 3 shows that the present evaluation of the use of computer assisted learning revealed statistically significant differences (p<0.01) for all six outcome measures. A massive effect size of 3.5 standard deviations (of difference between the experimental CAL group and the control group) emerged for the achievement outcome and an effect size of 1.4 standard deviations was found for the attitude measure. These differences favoured the CAL group. Also significant differences emerged in the students' perceptions between the computer and non-computer group for all of the four GCEI scales, namely, gender equity, investigation, innovation and resource adequacy. Table 3 reveals that the use of computer assisted learning was associated with an effect size of 1.0 standard deviations for gender equity, 1.9 standard deviations for investigation, 1.7 standard deviations for innovation and 1.5 standard deviations for resource adequacy. These differences favoured the CAL group in every case. Relative to control classes, the CAL classes were perceived to have greater gender equity, investigation, innovation and resource adequacy. The present study's findings in the context of computer assisted learning are consistent with past studies on non-CAL classroom environments (Walberg, 1975; Fraser, 1979, 1981, 1986) in which learning environment measures have proved useful in curriculum evaluation. Discussion The present evaluation of computer assisted learning in school geography education in Singapore is distinctive because, first, the CAL modules were developed specifically for the Singaporean context which required them to be content-specific and curriculum-specific, second, research on CAL in geography education in Singapore hitherto has been non-existent and, third, the classroom environment formed an important focus in the evaluation. In fact, the present research extended learning environment work in a new direction by developing and using a new instrument for assessing student perceptions of CAL classroom environments. This study suggests that appropriately-designed CAL courseware can be an effective instructional method in the classroom milieu, and that the educational application of appropriately-designed CAL courseware in social science classrooms has the potential to be efficacious. Hopefully, this study will serve as a catalyst for further research into the use of CAL courseware in the social science classroom. No systematic attempts previously have been made to examine the effects of CAL on learning environment characteristics. Little has been studied or is known about the impact of microcomputers on learning climates in education (Ellett, 1986). This study responds to the plea by Ellett (1986) Teh and Fraser 63 and Lancy (1987) to fill the lacunae represented by the study of computers and their impact on students. This study is significant because, in contrast to previous research, it uses courseware developed by the researchers in investigating computer learning environments in schools. One of the study's major contributions is that a new classroom environment instrument has been developed and validated specifically for the unique setting of computer assisted learning. The scales in this instrument display adequate factorial validity, internal consistency reliability, discriminant validity and predictive validity (in terms of being significantly related to student outcome scores). Also each scale differentiates significantly between the perceptions of students in different classrooms. It is likely that other researchers will find this new instrument useful in future studies of CAL classroom environments. The major finding was that, in contrast to past research, the use of CAI in this study led to a large impact in terms of achievement (effect size of 3.5 standard deviations), attitudes (1.4 standard deviations) and classroom environment (ranging from 1.0 to 1.9 standard deviations). The large effect sizes arising from this study could be attributed to the peculiar situation inherent in the Singapore education system. That is, the highly meritocratic, technologically-biased, centrally-controlled and achievement- oriented system might produce students who perform well. The large effect sizes seem to suggest that appropriate computer-based teaching can be effective with slow learners (ie., the Normal students), a finding which is consistent with the meta-analysis of CAL effectiveness reviewed by Kulik and Kulik (1991). However, there is a need for replication because of the uniqueness of the Singapore milieu and because results for slow learners might not be generalisable to other students. The specific content area for this study, namely, the topic of decision making in geography, was selected because practitioners (the teachers and principals) felt that this topic was the hardest to teach to slow learners. Furthermore, in accordance with the scheme of work prepared by schools, this topic would have to be taught at the time when the study was implemented. Because of this purposive and specific content area selected and its limited scope, the results of this study should be generalised with caution to other geography topics at the Secondary Two level. The findings of this study reflect the effectiveness of the present topic on decision making in geography; other CAL programs implemented under different conditions or using different geographical topics might not lead to the same results. There is considerable scope to make use of the new instrument for computer assisted learning environments in replicating the present evaluation of innovations in CAL, as well as in investigations of the effects 64 Australian Journal of Educational Technology, 1994, 10(1) of CAL classroom environments on student outcomes. In addition, it is hoped that researchers will make use of the new questionnaire specifically in CAL settings in pursuing some of the other lines of classroom environment research reviewed by Fraser (1994). These include the use of classroom environment research in school psychology (Burden & Fraser, in press), person-environment fit investigations of whether students achieve better in their preferred classroom environment (Fraser & Fisher, 1983), practical attempts to improve classroom setting (Fraser & Fisher, 1986), studies of links between classroom-level and school-level climate (Fisher & Fraser, 1993) and research which combines qualitative and quantitative methods in the study of learning environments (Fraser & Tobin, 1991). References Bangert-Drowns, R. L., Kulik, J. A. & Kulik, C. C. (1 985). Effectiveness of computer-based education in secondary schools. Journal of Computer- Based Instruction, 12(3), 59-68. Burden, R. L. & Fraser B. J. (in press). Use of classroom environment assessments in school psychology: A British perspective. Psychology in the Schools. Ellett, C. D. (1986). Conceptualizing the study of learning environments. In B. J. Fraser (Ed.), The Study of Learning Environments, (pp.34-40). Salem, OR: Assessment Research. Fisher, D. L. & Fraser, B. J. (1993). Interpersonal teacher behavior style and school environment. In Th. Wubbels & J. Levy (Eds.), Do You Know What You Look Like: Interpersonal Relationships in Education, (pp. 103-112). London: Falmer Press. Fraser, B. J. (1979). Evaluation of a science-based curriculum. In H. J. Walberg (Ed.), Educational Environments and Effects: Evaluation, Policy, and Productivity, (pp. 218-234). Berkeley, CA: McCutchan. Fraser, B. J. (1981). Learning Environment in Curriculum Evaluation: A Review. Evaluation in education: An international review series. Oxford, England: Pergamon Press. Fraser, B. J. (1986). Classroom Environment. London: Croom Helm. Fraser, B. J. (1994). Research on classroom and school climate. In D. Gabel (Ed.), Handbook of Research on Science Teaching and Learning, (pp. 493- 541). New York: Macmillan. Fraser, B. J. & Fisher, D. L. (1983). Student achievement as a function of person-environment fit: A regression surface analysis. British Journal of Educational Psychology, 53, 89-99. Fraser, B. J. & Fisher, D. L. (1986). Using short forms of classroom climate instruments to assess and improve classroom psychosocial environment. Journal of Research in Science Teaching, 23, 387-413. Teh and Fraser 65 Fraser, B. J. & Tobin, K. (1991). Combining qualitative and quantitative methods in classroom environment research. In B. J. Fraser and H. J. Walberg (Eds.), Educational Environments: Evaluation, Antecedents and Consequences, (pp. 271-292). Oxford, England: Pergamon Press. Fraser, B. J. & Walberg, H. J. (Eds.) (1991). Educational Environments: Evaluation, Antecedents and Consequences. Oxford, England: Pergamon Press. Fraser, B. J., Walberg, H. J., Welch, W. W. & Hattie, J. A. (1987). Syntheses of educational productivity research. International Journal of Educational Research, 11, 145-252. (whole issue) Hattie, J. & Fitzgerald, D. (1987). Sex differences in attitudes, achievement and use of computers. Australian Journal of Education, 31(1), 3-26. Kulik, C. C. & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7(1-2), 75-94. Lancy, D. F. (1987). The message is the medium: Studies of computer learning environments in schools. In B. J. Fraser (Ed.), The Study of Learning Environments, (Vol. 2, pp. 64-71). Perth: Curtin University. Low, K. G. (1988). Investigation of various modes of computer-based learning for secondary schools. Singapore Journal of Education, 9(2), 27- 32. Moos, R. H. (1974). The Social Climate Scales: An Overview. Palo Alto, CA: Consulting Psychologists Press. Moos, R. H. & Trickett, E. J. (1987). Classroom Environment Scale Manual, (2nd ed.). Palo Alto, CA: Consulting Psychologists Press. Ong, S. T. & Lee-Leck, M. K. (1986). The effects of computer assisted instruction on attitudes and achievement in mathematics on preservice primary school teachers. Selected Proceedings of Educomp '86: Impact of Microcomputers in Education - Issues and Techniques, (pp. 196-201). Singapore: Ngee Ann Polytechnic. Plomp, T. & Pelgrum, W. J. (1991). Introduction of computers in education: State of the art in eight countries. Computers and Education, 17(3), 249- 258. Roblyer, M. D., Castine, W. H. & King, F. J. (1988). Assessing the impact of computer-based instruction: A review of recent research. Computers in the Schools, 5(3/4), 1-149. Schubert, J. G. (1986). Gender equity in computer learning. Theory into Practice, 25, 267-275. Sutton, R. E. (1991). Equity and computers in the schools: A decade of research. Review of Educational Research, 61(4), 475-503. Teh, G. P. L. & Fraser, B. J. (1993). A study of computer assisted learning environments in Singapore. Paper presented at annual meeting of American Educational Research Association, Atlanta, GA. 66 Australian Journal of Educational Technology, 1994, 10(1) Walberg, H. J. (1975). Educational process evaluation. In M. W. Apple, M. J. Subkoviak & H. S. Lufler (Eds.), Educational Evaluation: Analysis and Responsibility, (pp. 237-268). Berkeley, CA: McCutchan. Welch, W. W. & Walberg, H. J. (1972). A national experiment in curriculum evaluation. American Educational Research Journal, 9, 373-383. Woo-Tan, L. B. (1989). Effects of Computer assisted instruction on the learning of transformation geometry. Unpublished Master's dissertation, National University of Singapore, Singapore. Dr George Teh is Senior Lecturer at Singapore's Nanyang Technological University's National Institute of Education. The work described here was part of his doctoral research at Curtin University of Technology, Perth. Dr Barry Fraser is Professor of Education and Director of the national Key Centre for School Science and Mathematics at Curtin University of Technology, Perth. Teh and Fraser 67 APPENDIX MODIFIED GEOGRAPHY CLASSROOM ENVIRONMENT INVENTORY (GCEI) ACTUAL FORM DIRECTIONS This questionnaire contains statements about practices which take place in this classroom. You will be asked how often each practice takes place. There are no 'right' or 'wrong' answers. Your opinion is what is wanted. Please do not write on this booklet. All answers should be given on the separate Answer Sheet. Think about how well each statement describes what actual classroom is like. Draw a circle around: 1 If the practice actually takes place ALMOST NEVER 2 If the practice actually takes place SELDOM 3 If the practice actually takes place SOMETIMES 4 If the practice actually takes place OFTEN 5 If the practice actually takes place VERY OFTEN Be sure to give an answer for all questions. If you change your mind about an answer, just cross it out and circle another. 68 Australian Journal of Educational Technology, 1994, 10(1) Please cite as: Teh, G. P. L. and Fraser, B. J. (1994). An evaluation of computer assisted learning in geography in Singapore. Australian Journal of Educational Technology, 10(1), 55-68. http://www.ascilite.org.au/ajet/ajet10/teh.html