Comparison of the cost-effectiveness of a computer-assisted learning program with a tutored demonstration to teach intestinal motility to medical students H. L. Leathard* and D. G. Dewhurst** * Department of Health Studies and Human Science, St. Martin's College, Lancaster ** Faculty of Health and Social Care, Leeds Metropolitan University Computer-based simulations of undergraduate experiments in pharmacology and physiology may offer a cost-effective alternative to the traditional live laboratory for some students, for whom laboratory skills are less important. Here we describe a study which compares two approaches to teaching preclinical medical students the pharmacology of colonic motility. Half of one cohort received a tutored live demonstration of an isolated tissue laboratory, while the other half used a computer simulation program covering the same subject. The study demonstrated that student learning was comparable for both groups, that many students found the computer simulation an acceptable alternative and that the latter required significantly less resource. Introduction The computer-assisted learning program Intestinal Motility was produced to help preclinical medical students to understand the variety of ways in which drugs or endogenous mediators can alter intestinal muscle movements and thereby influence propulsion of luminal contents. It simulates experiments actually carried out on a rat colon preparation, but the principles apply equally well to the small intestine and to the stomach and oesophagus of rat or other species. It can be used alone as a learning aid or can be used to support practical class teaching. The program consists of four discrete sections: Introduction, Method, Demonstration and Experiments, and the accompanying work booklet includes a set of student assignment questions together with model answers for tutors. Introduction contains a description of the objectives of the program, followed by an illustrated guide (a combination of text and colour graphics) to the general structure and innervation of the intestines. Methods uses a similar combination of text and graphics to describe the preparation itself, the experimental set-up and the apparatus used. Demonstration uses 118 ALT-} Volume 3 Number I animated graphics to show a physiological peristaltic response elicited by a brief, suprathreshold increase in intraluminal pressure and responses to intraluminal injections of test agents such as acetylcholine and adrenaline. This section is intended to give a visual impression of how the colon is seen to behave when the experiment is performed live. The main part of the program, Experiments, allows investigation of the direct effects of, and alterations in, the reflex response caused by a range of substances when applied either alone or in combination e.g. saline (control), neostigmine, atropine, acetylcholine, carbachol, phenolphthalein, adrenaline. The program has been demonstrated to the British Pharmacological Society (Dewhurst et al, 1992) Here we present data from a study designed to determine the usefulness of the computer program (CAL) as a stand-alone method evaluated by comparison with the conventional teaching method - a tutored demonstration (DEM)) using students for whom the program was originally designed, i.e. second-year preclinical students at the Charing Cross and Westminster Medical School in London. A brief account of part of the evaluation has also been presented to the British Pharmacological Society, but published only as an abstract (Leathard et al, 1994). Method The 1992-93 practical class 'circus' cohort of 156 students were randomized to one often groups, five DEM and five CAL groups (n = 14-16), for their intestinal motility practical. Each group received a common introduction in which the purpose and procedures of the evaluation were explained. Students were requested not to confer with members of the other group before handing in their work (on the fourth day after each class was held). They were assured that although their answers would be scored for the purpose of the evaluation, the scores would not contribute to their course-work mark (which would be determined separately), and that tutorial support, if needed, would be available prior to the practical course assessment. The completed assignments which were handed in by the appropriate deadline were photocopied. The originals were corrected and returned to the students. The unmarked copies were retained and scored for the evaluation exercise. Five of the twenty questions answered by the students were scored (Table 1). These were selected to test the following: students' extant knowledge; their understanding of the tissue preparation used; their data-interpretation ability; and the extent of their progress through a common sequence of tests. Both the DEM and CAL groups each had a tutor present throughout the 90-minute class, and tutors were randomized to take DEM and CAL classes on equal numbers of different occasions so that their individual characteristics should not influence the outcome of the evaluation. Students in both groups were advised to work through the various tests in the sequence given in the practical schedules that were provided, but the CAL group were not constrained to use this sequence. This study compares: • the resources used for the two teaching methods; 119 H. L Leathard and D. G. Dewhurst Comparison of the cost-effectiveness of a computer assisted learning program Table I: Comparison of the learning scores of students from the two groups ¡n the study. The DEM group (n = 52) received a tutored demonstration of a live laboratory practical and the CAL group (n = 53) received a computer simulation which allowed them to access typical results from a number of experiments performed on this preparation. Student learning was assessed using an assignment comprising twenty questions which were selected to test the following: students' extant knowledge; their understanding of the tissue preparation used; their data interpretation ability; and the extent of their progress through a common sequence of tests. For this study five of the questions answered by the students were scored. Learning scores are expressed as the percentage of students giving satisfactory answers to the questions. Question QI Explain the response elicited by the control injection of saline. Q7 The cone, of acetylcholine injected into the lumen to stimulate longitudinal muscle in this preparation is 10,000 times the concentration required in a conventional organ bath. What are the possible explanations? Q11 Explain the mechanism of action of neostigmine and suggest whether cholinesterases in this preparation will remain inhibited for (a) minutes, (b) hours, or (c) days. Q14 Explain the recorded flow pattern (in the presence of phenolphthalein) in terms of circular muscle activity. Q20 Suggest ways in which the atropine-resistant component of the stimulant effect of phenolphthalein could be mediated. DEM 95 70 86 63 34 CAL 98 50 72 51 19 Mean 97 61 78 56 26 • the students' learning; • the attitudes of students from both groups to pertinent aspects of computer-assisted learning. The attitude assessment was based on a slightly modified version of the questionnaire used previously by Dewhurst et al. (1994). The first three questions sought information about the students, access to and familiarity with computers. Twenty-five statements about pertinent aspects of computer-assisted learning were then given, and students were asked to indicate their level of agreement/disagreement with each statement using a five- point Likert scale (strongly agree; agree; undecided; disagree; strongly disagree). A final open question invited any further comments the student might wish to make. Results I Student learning One hundred and twelve students submitted their answers by the stringent deadlines set for the evaluation. Of these, fifty-two were from the CAL group and fifty-three from the DEM group, with seven failing to indicate the group they had belonged to. Their learning 120 ALT-J Volume 3 Number I scores are shown in Table 1. While virtually all students gave satisfactory answers to the first question, subsequent questions were progressively less well answered, and the DEM students performed slightly better than the CAL students in each case. There was no statistically significant difference in the mean scores of the two groups. 2 Resources As detailed in Table 2, the human resources used for the DEM students totalled fifty-six hours of technician's time and twenty-four hours of tutor's time. Equivalent figures for the CAL students were four hours and fifteen hours respectively. Conservatively, these hours could be costed at £10 per hour, in which case the staff costs for the DEM groups would be £800, and for the CAL groups £190. The costs of materials were £130 for the CAL groups and £60 for the DEM groups. It should be noted that the CAL costs are a Table 2: Comparison of the resources (staff and materials/consumables) required for the two groups in the study. The DEM group were taught using a tutored demonstration of the live experiment in the laboratory, while the CAL group used a computer simulation of the experiment Staff resources are measured in hours of staff time while materials are costed in £ Sterling. The equipment required for both approaches has not been considered since it was available from stock Resources 1 Staff Resources (i) Technical staff support advance preparation for class preparation on days of class (3 h on each of 5 days) technical support during class clearing up after class (ii) Demonstrators (post-graduate students) advance preparation class supervision Total 2 Consumables/Materials drugs, solutions, gases rats computer program - Intestinal Motility (reusable) diskettes (reusable) Total DEM 16h 15h 15h 10h 4h 20h 80h £10 ' £50 0 0 £60 CAL 2h 0 0 2h 0 15h 19h 0 0 £120 £10 £130 121 H. L Leathard and D. G. Dewburst Comparison of the cost-effectiveness of a computer assisted learning program one-off expenditure since the program and disks may be used again. Apparatus for both teaching methods was available from stock and has not been considered. 3 Student attitudes One hundred students returned the Attitudes questionnaire, of whom 40 belonged to the CAL group, 48 to the DEM group; 12 did not indicate to which group they had belonged. Of these, 26 owned a computer, 82 stated that they had access to a computer, and 69 had some (albeit limited) previous experience of using a computer program as an alternative to performing a practical using live tissue (questions 1-3 on the questionnaire). In general, the views of students from each group were similar, so the main part of this analysis is based on the responses of all the 100 students, points where there was some divergence being addressed afterwards. In the following summary, where responses are grouped, the number of students giving a particular response is indicated in parentheses after the response. Very small numbers of students responded 'strongly' to any of the statements. More than two-thirds of the respondents agreed or strongly agreed that 'Computer simulations enable the student to control the pace of learning' (93), 'Computer simulation programs allow acquisition of accurate results in the allocated time' (85) and 'Students who need laboratory skills for their future careers can only obtain these skills by performing practical experiments on live tissues' (75); and similar proportions disagreed or strongly disagreed that 'There is no place for computer simulations of laboratory practicáis in undergraduate bioscience degree courses' (91), 'Computers are difficult to use, and should be avoided when possible' (86), 'AH undergraduate experiments using live tissues should stop' (82), 'Computer simulations of practical experiments are of very limited value in the teaching of practical physiology and pharmacology' (69) and 'Practical laboratory work on live tissues is the only successful method of reinforcing the theory of physiological principles' (68). Between half and two-thirds of the students agreed or strongly agreed that 'An important bonus in laboratory practicáis is that teacher-student and student-student contact is encouraged and learning is thereby enhanced' (66), 'Undergraduate laboratory-based practicáis are more expensive to carry out than utilizing computer simulation programs' (54), 'Replacing laboratory-based experiments with computer simulations provides the lecturer with more time for lesson planning, marking work, etc' (52) and 'During laboratory practical experiments, the time spent learning how to use equipment often leaves insufficient time to complete the experiment' (52). Similar proportions of students disagreed or strongly disagreed that 'Students need to receive special training to use a computer simulation' (62), 'Students always get poor results from their laboratory practical experiments' (59), 'Computer simulations of laboratory experiments can lead to the acquisition of laboratory skills' (56) 'The use of live tissue to demonstrate physiological/pharmacological principles poses no moral dilemma' (53) and 'Computer simulations of such experiments are difficult to use' (51). Between one-third and a half of the students agreed or strongly agreed that 'The use of computer simulations of practical experiments enables the lecturer to spend more time 122 Volume 3 Number I with the students' (48), 'Hands-on experience with live tissues is essential for all students of medicine' (45), 'There is a place for computer simulations only if they are used as a supplement to demonstrations so that the real experiment can be seen first' (45) and 'Computer simulations of experiments on live tissues can replace laboratory practical experiments as they meet the majority of course objectives (45). Similar proportions of students disagreed or strongly disagreed that 'Computer simulations of practical experiments are less instructive than laboratory-based practicáis' (49), 'Computer simulation programs cannot adequately replace laboratory-based practical experiments in the teaching of physiological and pharmacological principals' (46), 'Computer-based teaching is boring, thus limiting student learning' (45), 'Computer simulations convey little sense of the "real" experiment, therefore all appreciation is lost' (40), 'Hands-on experience with live tissues is essential for all students of medicine' (34) and 'During practical experiments the time spent learning how to use equipment often leaves insufficient time to complete the experiment' (34). The highest proportions of'undecided' responses were elicited by the statements that 'Undergraduate laboratory-based practicáis are more expensive to carry out than utilizing computer simulation programs' (40), 'Replacing laboratory-based experiments with computer simulations provides the lecturer with more time for lesson planning, marking work, etc' (37) and 'Computer simulations of such experiments are difficult to use' (34). There were some substantial differences in the proportions of responses to a few of the statements by students from the CAL and DEM groups. In the following summary the proportions are given as percentages of the forty CAL students and forty-eight DEM students for ease of comparison. Whereas 65% of the DEM group felt that use of computer simulations freed lecturers' time, only 43% of the CAL group expressed this view. Forty percent of the CAL group but 17% of the DEM group concurred with the statement about computer simulations being less instructive than the real thing, whereas 15% and 38% respectively were undecided. Also, 45% and 65% respectively of CAL and DEM groups agreed that laboratory experiments are more expensive than computer simulations, but 50% and 29% respectively were undecided. Discussion Many of the results of this evaluation are unequivocal and need no further comment. The computer simulation class required significantly less staff resource (hours of technicians' and tutors' time) than did the live demonstration, and can therefore be considered to be less expensive since other costs were comparable. The overall cost-effectiveness depends not only on actual costs but also on the adequacy of student learning. In this respect the CAL group performed less well than the DEM group although this varied from question to question. Question 1 was based on a fundamental concept that was explained carefully, and related to the first of the common sequence of tests. It is entirely appropriate that almost all the students gave a satisfactory answer. In contrast, question 20 was based on the last section of the sequence of tests and involved concepts that were relatively novel to the students. Consequently, the numbers 123 H. L Leathard and D. G. Dewhurst Comparison of the cost-effectiveness of a computer assisted learning program giving satisfactory explanations were low. Most of those in the CAL group reached this stage within the time allocated, whereas the demonstration of the live practical did not regularly progress this far. To compensate for the incomplete experiment, however, the tutor explained the final parts with the aid of a sample tracing from a preVious experiment (all students were provided with these). Thus, it seems that the tutors' explanations and the integral discussion enhanced the students' understanding of the challenging parts. From students' comments and tutors' observations, however, it is evident that CAL students needed an opportunity for discussion with a tutor after completing the program, not while they were concentrating on the screen; so tutors' time was not used optimally for this teaching method during the evaluation. The students' attitudes were generally favourable to computer simulations of experiments but, from the responses to the open question and the views summarized above, it seems that the majority would like to have some experience of the live practical with computer simulations to support this. Clearly in a financial climate where the unit of resource available for teaching and learning is being reduced, teachers must consider carefully the costs of certain teaching styles relative to the learning objectives. In this study it might be argued that, for medical students, a live demonstration is unnecessary since it is the knowledge and understanding of the pharmacology of colonie motility which is the important learning objective rather than the laboratory skills associated with the investigation. In this study student learning, as measured by answers to key questions, is comparable for both groups, while the cost of the live laboratory is significantly higher. The findings of this study are similar to those of similar studies where computer-based learning methods have been compared to traditional live laboratories in undergraduate physiology and anatomy (Dewhurst et al, 1994; Coleman et al, 1994; Guy & Frisby, 1992; Fawver et al, 1990). Acknowledgement We wish to thank Professor Julia Buckingham and her staff in the Department of Pharmacology at the Charing Cross and Westminster Medical School in London for their support of and contributions to this evaluation. References Coleman, I., Dewhurst, D.G., Meehan, A.S. & Williams, A.D. (1994) 'A computer simulation for learning about the physiological response to exercise', American Journal of Physiology, (Advances in Physiology Education), II, S2-S9. Dewhurst, D.G., Leathard, H.L., Higman, M. and Ullyot, R. (1992), 'A computer simulation to teach the action of drugs on intestinal (colonic) motility', British Journal of Pharmacology, 106, 143. Fawver, A.L., Branch, C.E., Trentham, L., Robertson, B.T., and Beckett, S.D. (1990), 'A comparison of interactive video instruction with live animal laboratories', American Journal of Physiology, 259 (Advanced Physiological Education, 4), S11-S14. 124 Volume 3 Number I Guy, J.F. & Frisby, A.J. (1992), 'Using interactive videodiscs to teach gross anatomy to undergraduates at the Ohio State University', Acad Med, 67, 132-133. Leathard, H.L, Cover, P.O., Dewhurst, D.G., Kumari, M. and Rantle, C. (1994), 'Evaluation of a computer-assisted learning program on intestinal motility', British Journal of Pharmacology, 112, 179. 125