The use of computers as substitute tutors for marketing students Miriam Catterall and Patrick Ibbotson School of Management, University of Ulster The use of computers as substitute tutors is associated primarily, though not exclusively, with multiple-choice question formats (Ellington, Percival and Race, 1993). We report the findings of a project that involved the design, testing and evaluation of a set of computer-based tutorials employing multiple-choice questions with 700 students on postgraduate and undergraduate introductory Marketing modules. The computer-based tutorials were designed to meet two main objectives, namely to help students in their formative assessment and to help staff monitor any difficulties students were experiencing with module content. However, students incorporated the tutorials into their learning in ways that had not been anticipated specifically; they used the tutorials for a number of related but different purposes, and their usage patterns varied considerably. Introduction The importance of feedback in the learning process is self-evident and well documented (Rowntree, 1987; Brown & Knight, 1994). Students tend to place high value on the feedback they obtain from teaching staff through the assessment process. Assessment can consume as many staff hours as teaching (Gibbs & Jenkins, 1992). With increasing class sizes, teaching staff are encouraged to find ways to reduce the demands the assessment process places on their time. Peer and computer-marked assignments can help in this respect. Our aim was to develop a Computer-Based Tutorial (CBT) that would provide students with early and frequent opportunities to assess their own progress. The CBT contains eleven separate tutorials corresponding to the main topic areas covered on the Introduction to Marketing module. When students select a tutorial, they are presented, with twenty-five multiple-choice questions (MCQs) selected at random from a bank of over fifty. Students have two chances to answer each question, and if they still answer incorrectly they are told the correct answer and directed to the appropriate page in the text (see Figure 1). On completion of the tutorial, students are given their score and taken 86 A I T J Volume 3 Number I If Uonfiir, a hairdryer Manufacturer« is focusing on custoner satisfaction, it uill find that what corauners really want i Figure /: Typical MCQ from the test bank back to the main menu where another choice can be made. Students are able to access the CBT on campus machines or, if they have a machine at work or at home, a take-away version is available on floppy disk. A pre-pilot study with fifty-seven students during the 1993 summer semester revealed that students were using the CBT more frequently and for longer periods of time than was anticipated. Their recommendations for improving it suggested they were using it for reasons other than self-assessment. The main pilot was undertaken during the winter and spring semesters of 1993-94, during which time over 700 students had been given access to the CBT. Log files were used to capture data on the frequency and patterns of CBT usage on both on- and off-campus machines. Feedback from a sample of student users was obtained by structured questionnaire, unstructured interviews with individual students, projective techniques and direct observation. The main pilot confirmed earlier findings, and these are reported in detail elsewhere (Catterall & Ibbotson, 1994). In the remainder of this paper, we discuss the findings of both the pre- and main pilot studies, and present our understanding of how and why the CBT is incorporated into the student learning process. Main findings Part-time students are the heaviest users Part-time students were more likely to use the CBT, use it more frequently and for longer periods of time. For example, single sessions of up to one hour were not uncommon among part-time students. 87 Miriam Caaerall and Patrick Ibbotson The use of computers as substitute tutors One of the reasons for this difference was the availability of the CBT on a take-away disk for use with off-campus computers, at home or at work. While under 10% of full-time students obtained take-away disks, over 60% of part-time students did so. However, part- time students without access to off-campus machines were more likely to use the CBT than their full-time counterparts, even though they are on campus for considerably fewer hours per week. Use decreases over time Use of the CBT decreased over time among all groups of students including part-timers and those with access to off-campus machines. To illustrate: 74% of a group of full-time undergraduates used the CBT regularly over the first half of the semester; during the second half, only 32% were still regular users. There are a number of possible explanations for this finding: MCQ Assessment. Half of the groups or classes who participated in the pilot study were given an MCQ test (pen and paper) as one of their coursework assignments. This MCQ test was administered at the halfway point in the semester. It is reasonable to assume that a number of CBT users were practising for this test, and on its completion would no longer feel the need to use the CBT. However, since CBT usage decreased also in groups that were not assessed by MCQ test, this is not a complete explanation. Novelty. Very few students who participated in the pilot had any previous experience of this type of computer-based learning. It could be hypothesized that a new and different learning resource will hold some novelty value for students that will wear thin after a few use-experiences. Evidence to support or reject this is not immediately apparent. We did find some evidence that newer learners were more likely to perceive the CBT as a useful learning resource than more experienced learners. Full-time postgraduate groups (experienced learners) were the least likely to use the CBT. A number of students in these groups reported that 'real' study involves texts and journal articles, and that in their perception CBTs are of marginal value. By contrast, students new to the university environment (first-year undergraduates) and those returning after a break from formal education (part-time postgraduates) were more likely to use the CBT and look upon it as part of the total package of learning resources. One group of part- time undergraduates who studied Marketing as their first module assumed that other modules would also be offering CBTs as part of the learning resources, and were surprised and disappointed to find this would not be the case. The need for support reduces? On the basis of the evidence we have on how students use the CBT and the reasons why they use it, it appears reasonable to conclude that the need for the type of learning support provided by the CBT will decrease over time. Students used the CBT in a variety of ways and for,/different reasons, including: Preparation for learning. This involves working through a tutorial as quickly as possible (any answer will do) to identify the subject matter of the questions and the concepts or ALT-] Volume 3 Number I learning points that are questioned most frequently. This gives the student clues as to which concepts or issues are most important and where effort needs to be concentrated. This is likely to be more important for students in the early weeks of a new module when every bit of information seems vital. As the weeks progress, they may be better able to differentiate between key learning points and interesting information during lectures. Confidence building. The student has not read the lecture notes or the text, and works through the CBT trying to answer the questions correctly. Good luck or common sense will ensure that they get some answers correct. Students report that this gives them a sense of achievement and confidence. In other words, they already know something about this subject even before they begin to study it. They report that this makes the study of text and notes a less daunting prospect. This is also likely to be more important to the student during the first weeks of a new module. As they learn more about the subject and become familiar with the text, they may need this type of CBT support less frequently, if at all. Studying. The student has not studied the lecture notes or the text, though he/she may have read them through once or twice. The student works through the CBT with the notes and text to hand, answering the questions and checking the appropriate reference in the textbook. As the weeks progress, and the student becomes more familiar with the material, the need for this type of support should diminish. This type of studying appeared to be very time-consuming though many students continued to use it throughout the semester. When asked why, the most frequent responses were that they preferred to learn this way and it helped them concentrate. Self assessment. After having studied the material, students use the CBT to assess their learning. When asked about the benefits of this approach, they reported that it gave them a sense of achievement and confidence. Part-time students reported that it was important to feel they had achieved something early on in the module. In other words, in the first few weeks they were uncertain as to whether they had made the right decision in returning to study: would they be able to study, were they studying enough or too little? The early feedback from the CBT provided reassurance in these respects. Again, it could be argued that as students' confidence increases, the need for this kind of CBT support reduces. Drawbacks with heavy CBT usage The design of the CBT was based on the following assumptions: • a typical CBT session would involve the student using a single tutorial; and • a single tutorial would typically take up to fifteen minutes to complete. The data reveals that there is no typical session or typical session-length. Some students would use five or more different tutorials or the same tutorial five or more times in a session. Very few users would undertake only one tutorial in a session. Tutorial times varied in length between four minutes (presumably for preparation for learning) to forty minutes or more (probably using with a textbook). 89 Miriam Catlerall and Patrick Ibbotson The use of computers as substitute tutors Given the long duration of many individual sessions, it is not surprising that a number of students reported suffering from eye strain. It emerged also that question design can contribute to this; specifically, questions where the answer options were long and wordy did not work well. Ideally, when designing MCQs for the screen, the answer options need to be short and sharp. A number of students reported that the CBT was easy and fun in the beginning but became monotonous and boring in later weeks. This is understandable, given the very heavy use made of the CBT by these students in the first half of the semester. We have addressed this issue in two ways: More variety in question design. The MCQ format is flexible and offers many advantages over other question types (Ebel & Frisbie, 1986). The addition of graphics to illustrate questions should be an improvement on the current all-text format. The number of answer options is now more varied, ranging from two to five, when previously all questions had five options. The design of open questions will reduce the monotony of simply using the mouse to point to and select the chosen response. Progression. One of our original design specifications was that progression should be built into the CBT. Specifically, later tutorials should contain more difficult questions than earlier tutorials, and later tutorials should build on knowledge and comprehension of earlier tutorial material. Earlier tutorials do contain a higher proportion of easy questions, and this was identified by students. The proportion of difficult questions in later tutorials has been increased. Designing MCQs is acknowledged to be a time-consuming and difficult task (Osterlind, 1989). Designing MCQs sufficiently short and sharp for the screen that also build on earlier tutorial material is proving difficult to realize. Conclusions The most disturbing finding from the pilot study was that the number of users of the CBT and their frequency of use decreased over time. Further, some of the disadvantages identified by students give cause for concern; notably that it becomes boring and causes eye strain. However, it would be inappropriate to take these findings at face value and conclude that the CBT has failed to engage students sufficiently. The CBT was created with the overall aim of providing students with early and frequent opportunities to assess their progress on the Marketing module. In the event, self- assessment was only one of the reasons why students used the CBT. Students were as likely to use the CBT to help prepare them for learning and to build confidence as to self- assess what and how much learning had been achieved. In this respect, the CBT proved especially useful for part-time students who needed the reassurance in the first few weeks of the module that, under ideal circumstances, a human tutor would be able to provide on a one-to-one basis. As students gain confidence and familiarity with the subject matter and their study of it, the need for the kind of support the CBT provided will reduce. 90 ALT-J Volume 3 Number I The disadvantages identified by students need to be addressed. For example, if some students are experiencing eye strain from sitting for long periods in front of the screen, instructing them not to do so is unlikely to solve the problem. Similarly, if students get bored because they use the CBT far more frequently than was ever intended, telling them to use it less frequently is not a helpful response. To this end, we have attempted to introduce more variety into the question format, reduced the amount of text that appears on screen and made tutorials progressively more challenging. The pilot study revealed the likely demand among students for computer-based learning resources that can be used off the campus. The majority of part-time students have access to off-campus machines. There are a number of advantages in encouraging students to use CBT off-campus material, not least of which is that it reduces the demand for on- campus machines to the benefit of students without off-campus access. Marketers employ research to identify how consumers perceive and relate to products, since consumers will identify new uses and benefits not envisaged by manufacturers. Our pilot study illustrates this applies equally to CBT development. References Brown, S. and Knight, P. (1994), Assessing Learners in Higher Education, London, Kogan Page. Catterall, D.M. and Ibbotson, P.G. (1994), 'The development of a low technology marketing CBT', Account, 6, 1 (Spring). Ebel, R.L. and Frisbie, D.A. (1986), Essentials of Education Measurement, London, Prentice Hall. Ellington, H., Percival, F. & Race, P. (1993), Handbook of Educational Technology, London, Kogan Page. Gibbs, G. and Jenkins, A. (1992), Teaching Large Classes in Higher Education, London, Kogan Page. Osterlind, S.J. (1989), Constructing Test Items, Boston MA, USA, Kluwer Academic Publishers. Rowntree, D. (1987), Assessing Students: How Should We Know Them?, London, Kogan Page. 91