The Works Metallurgist: an evaluation of a CAL package on phase diagrams Canan Tosunoglu,* Philip Butcher;** Eileen Scanlon* and Ann Jones* institute of Educational Technology, The Open University **Academic Computing Service, The Open University The Works Metallurgist is used in the Open University course: Materials — Engineering and Science. It is being evaluated as part of a larger study of CAL teaching in Science and Technology at the Open University. With 540 students, the course provides a sufficiently large sample for a range of evaluation methods to be employed in a variety of settings (e.g. home, residential school). The methods used include questionnaires, attitude scales and pre/post achievement tests, observations and interviews. The focus of the evaluation exercise is on the effectiveness of CAL in contributing to student learning. The issue of the educational quality of the program and its instructional features are considered in this paper, in association with the goals of the program. Introduction A number of factors have converged today to force the development of more efficient and cost- effective educational strategies. These include rapidly changing technology, the increasing number of students in higher education, and resource restrictions imposed on all sectors of education. Educational systems are therefore finding it increasingly difficult to respond to learners' needs, highlighting the need for alternative educational strategies. An increase in the educational use of computers and new information technologies is often one of the suggested alternatives, baised on the premise that computers can support student learning in an individualized way and at the learner's own pace. In this context, evaluation in general and CAL evaluation in particular can contribute to the process of defining and refining educational strategies. The Open University (OU) has included the use of computers to support learning in its courses since 1971, and since 1979 attempts to evaluate the OU's CAL provision have been reported in a number of publications. Despite a large volume of research into CAL evaluation, we believe that there is a need to develop a more comprehensive and systematic approach which can contribute more directly to the development of improved CAL provision. 51 Canon Tosunoglu et al The Works Metallurgist an evaluation of a CAL package on phase diagrams Aims of the evaluation study One purpose of this evaluation was to contribute to decision-making on the amount of CAL materials in courses. More specifically, our aim was to determine whether a particular CAL program was effective in contributing to students' learning about phase diagrams. Through a series of studies, we aim to build up a more general knowledge of what features contribute to effective CAL in order to develop and improve other educational software. Our study aimed to answer the following questions: • How do students use the program? • Does the software contribute to student learning? • What do students learn from the program? • How can the program be improved? Our main concern is the learning benefits to students, and the relatively small size of the program helped us to investigate a number of aspects of the program in relation to learning, and the extent to which the program's objectives were attained. Fortunately, the objectives of this program were already well defined, as it was developed in response to students' difficulties in interpreting phase diagrams. Phase diagrams A phase diagram maps the ranges of composition and temperature over which particular phases are stable in a given system of materials. They enable the student to anticipate the alloys that are likely to have desirable properties, the treatments that can be used to develop best properties, and the treatments that should be avoided (Newey and Weaver, 1990). An understanding of phase diagrams is an important part of chemistry and related curricula (Williams and Collins, 1994), although the coverage of the topic in many textbooks has been criticized (Kildahl, 1994). The phase diagram used in the Works Metallurgist is a binary system consisting of tin and lead, and it has commercial importance as soft solders. Software: The Works Metallurgist The Works Metallurgist is an interactive tutorial designed to teach interpretations of phase diagrams and the Lever Rule (a method of calculating percentage of solid and liquid in an alloy at a given temperature from a phase diagram) to Open University students who are studying at a distance. It was developed for Materials - Engineering and Science, a second-level course (equivalent to second-year full-time study) which runs from February until October and includes a residential summer school. Course evaluations had shown that students had difficulties in interpreting and applying phase diagrams, and the program was specifically developed to help students in this area. It is in a game format, and students are given job titles ranging from Applicant for Apprentice Metallurgist to Works Metallurgist, according to their performance. The software is available for students to buy, and is one of five programs produced for the 52 Volume 4 Number I course. In order to run the programs, students need access to a computer with the OU's specification (basically, a PC) and either Windows or GEM. The package is also available at the course's residential school which provides another opportunity to use it. Use of The Works Metallurgist at residential summer school Students used the CAL programs during their laboratory work, both while waiting for parts of the practicals to reach the next stage, and when they finished the experiments early. Although the program was designed for individual use, students mainly worked in the laboratories in pairs, and usually chose to use the program in pairs too, and to discuss their answers with each other before typing them in. Three computers were also provided in the student hall for use at any time. Sample and design of study The population consisted of 540 students who were studying Materials: Engineering and Science in 1995. The researcher attended two weeks of residential school (out of seven) and conducted observations and interviews; The students had reasonable familiarity with computers, and were given a questionnaire, attitude scale, and knowledge pre-test, along with the evaluation disk (a special version of the program that recorded some usage information). Methods used We used a number of different approaches, and gathered different types of data. Our rationale for using these different methods is reported in Jones et al (in press). Observation Forty-four sessions were observed, with session times ranging from 3 to 53 minutes. Observations were carried out both in the laboratories and in the student hall. Students' inputs were recorded, and where appropriate the students were asked to supply reasons for their answers. Although the observer tried to minimize the disturbance to students' natural progress with the program, she was occasionally asked for help with the tasks, and this was used as an opportunity to ask probing questions. Questionnaires and evaluation disk A two-stage questionnaire was given to 60 students who were randomly selected from those with access to a computer. Stage 1 of the questionnaire concerned previous OU courses, study of phase diagrams, and the students' computing background. Students filled in this part of the questionnaire before they used the software in order to complete an attitude measure and a knowledge pre-test. Stage 2 concerned the efficiency of the program, the quality and nature of students' interactions with it, their difficulties, their opinions of the program, and general comments. These 60 students were given a special version of the program which recorded usage information and were asked to return the disk after about five weeks. This enabled us to see how much time students spent on the program, which sections took most time, and how much improvement on their understanding they made in that limited time. Attitude measure This was used to obtain a measure of students attitudes towards computers and learning with computers both before and after using the program, although the post-use questionnaire also 53 Canon Tosunog/u et ol The W o r t s Metallurgist an evaluation of a CAL package on phase diagrams contained items related to the program, and was to be completed by students along with the second part of the questionnaire at the end of the five-week period. Achievement measure Students completed a knowledge pre-test along with the first part of the questionnaire, and they were tested again (using the same questions) after they had finished working with the program. Four more short questions covered the important concept of phase. Interviews Formal and informal interviews were conducted in the laboratories and in the computer suite in the student hall. Tutorials related to phase diagrams were also attended by the researcher, and students' attendance and activities during the tutorial were observed. How do students use the program? Observations revealed that The Works Metallurgist was the program used most during the two summer-school weeks. Comparatively very little time was spent on other programs, and even students who had the program at home used it in the summer school. Most students who saw it for the first time decided to buy it, if they had access to an appropriate computer. These observations about its use and popularity are supported by the sales figures. Students reported that the program was very useful, and most could only suggest making minor changes to it. None of the students said the program was difficult to use, and they were easily able to use specific features of it such as Crosshairs, and the Draw Tie Line and Show Labels facilities.1 The use of the calculator was found to be somewhat tedious in Windows, but less so in the GEM version. Evaluation disks returned by students showed that they spent a great deal of time using the program, ranging from 12 to 276 minutes with a mean of 128 minutes. Most of the students spread their use of the program over a number of sessions. The average number of sessions was 3.6. Sections 1 to 4 require students to interact with the computer. There are also two further sections where students carry out the activity away from the computer. The overall results for sections 1 to 4 are given in Table 1. (The total is given for all students across the whole program. Mean, maximum and minimum values are per student for one section.) Time spent using the program (minutes) Errors made Sections 1-4 total 2440 mean 9.25 maximum 36 minimum I mean Z75 maximum 13 minimum 0 Table I: Analysis of returned evaluation disks (N = 25) Attitude scale and questionnaire Fifty percent of the questionnaires were finally returned and findings from the returned questionnaires are consistent with this positive view as can be seen in Table 2. 54 ALT-J Volume 4 Number I Strongly agree Agree Neutral Disagree Strongly disagree I I did not learn anything new from 19 9the program 2 The program is capable of teaching about phase diagrams 3 The program makes you try to get a higher job title 4 Using the Works Metallurgist assisted in my understanding of phase diagrams 5 The program is helpful in preparing for the exam 6 1 still would have preferred to learn about phase diagrams from a tutor 7 4 II 10 21 II 17 15 7 1 1 8 1 4 16 Table 2: Items from the attitude scale related to The Works Metallurgist (N = 29) Do students learn from the program? Students views related to the ease of use, value, learning, difficulty and motivational aspects of the program were positive, and the aim of the software was clear to every student. They also believed they had benefited: 'I think I have learnt more from this than reading the text book, it is very good, very good.' 'I need it.' 'Quite educational.' The one student who was negative - 'I don't like anything related to computers' - initially chose not to use the program, but when he did use it later, he said to the observer: 'It's not like using a computer, it's like a game'. Pre-test Post test Mean 3.82 3.91 Std Dev 1.06 1.08 N 28 23Table 3: Comparison ofpre- and post-test scores Observations supported their beliefs: students were observed to interpret phase diagrams and apply the Lever Rule correctly, although they were unsure about this procedure or unable to apply the rule correctly before using the program. However, the analyses of knowledge questions did not show any differences between the pre- and post-test scores (see Table 3). As can be seen from the sample size in Table 3, there was some reluctance to answer the post- test, and because the students were working at home, the timing and circumstances of the post-test were beyond our control. Nevertheless, the lack of difference between pre- and post- test findings is surprising in this study as observations as well as the students' own perceptions did suggest some benefit. However, there are a number of difficulties with the use of pre- and post-tests in CAL evaluation which are well documented (Jones et al, in press). Such 55 Canon Tosunoglu et ol The Works Metallurgist an evaluation of a CAL package on phase diagrams instruments are not sensitive to the complexity of the learning situation, and may not reveal anything about the learning process itself. Indeed, data from the evaluation disks clearly shows that the time spent and errors made in each section decreased with repeated use of the program. How can the program be improved? When asked about possible changes to the program, students suggested that it could be more flexible about spelling mistakes and routing of questions, and also include a Don't Know button. Students mentioned the following points when asked about difficulties: using the calculator (27%), finishing the tasks in one session (20%), understanding questions (20%), understanding explanations to wrong answers (20%), calculating the answers (13%), and using the mouse and crosshairs (6%). These findings again are consistent with observations. Fifty- three percent of students reported that when they had difficulties using the program, they repeated the program. The program can be further improved by small changes in question order, and wording to help students understand the questions better, and by providing help for students in deciding which side of the tie line represents what phase (i.e. solid or liquid), simplifying the operation of the calculator, clarifying the distinction between certain questions about solid or liquid composition, and allowing a little more flexibility with spelling mistakes. Conclusion Our overall findings showed that this program was perceived as very useful and necessary by students. They also stated that the game-like nature of the program, or getting it wrong, made them try again and again. This feature is very important because it shows that the program engages students, and as a result their chances of learning increases. The clear and simple interface of The Works Metallurgist is among the positive factors mentioned by students. We wanted to find out the extent to which students learn from the program. Our findings did not show any direct relationship between its use and student performance, but our indirect findings indicate that it can contribute to student learning. In summary, The Works Metallurgist has certain critical properties that help its success: • students think they learn with it; • it is enjoyable to use; • it does not require too much computer familiarity; • it is highly motivational (humour, game-like format). Note 1. 'Crosshairs' gives the temperature and composition of any point on the diagram when the cursor arrow is placed on it. 'Draw Tie Line' draws the tie line and gives the solid and liquid 56 ALT-J Volume 4 Number I composition of a given alloy. 'Show Labels' gives the names of the different regions and lines of the phase diagram. References Jones, A., Scanlon, E., Tosunoglu, C., Ross, S., Butcher, P., Murphy, P. and Greenberg, J., 'Evaluating CAL at the Open University: 15 years on', Computers and Education, (in press). Kildahl, N.K. (1994), 'Journey around a phase diagram', Journal of Chemical Education, 71 (12), 1052-5. Newey, C. and Weaver, G. (1990), Materials: Principles and Practice, London, Butterworth (in association with the Open University). Williams, K.R. and Collins, S.E. (1994), 'The solid-liquid phase diagram experiment: updated for the physical chemistry laboratory', Journal of Chemical Education, 71 (7), 617-20. 57