Using Web-based support for campus-based open learning: lessons from a study in dental public health Ruth Holt* Martin Oliver** and Claire McAvinia** *Department of Transcultural Oral Health, Eastman Dental Institute, UCL **Education and Professional Development UCL email: r.holt@eastman.ucl.ac.uk There has been much written about the use of the Web in higher education, much of which advocates its use as an effective way of supporting learning, particularly in terms of the desirability of features such as flexibility and the value of online discussions. In this paper, a case study is described which calls some of this received wisdom into question. The study also explores wider issues of curriculum design, particularly in terms of the role of assessment and of self-assessment, both of which played a crucial role in the course. Unlike many studies, then, the purpose of this paper is not to demonstrate the success of a particular approach or to advocate particular forms of practice, but instead to highlight the shortcomings of existing guidelines for curriculum development in this area. This suggests that further inquiry into this form of education is required — and in particular, inquiry that pays detailed attention to the backgrounds of learners, and involves close study of their experiences. Introduction The use of the Web to support courses in higher education has become commonplace in recent years. However, much of what has been written has concentrated on the features of the medium that make it suitable for distance learning; there has been less written about the influence of these features on campus-based courses. This paper presents a case study which explores this particular problem, and which questions received wisdom about some of the claims that the pedagogic features of Web-based learning are desirable for learners. Firstly, however, the literature on campus-based open learning, and on the use of the Web to support dental education (which forms the focus for this study) will be reviewed. 51 Ruth Holt Martin Oliver and Oaire McAvinia Using Web-based support for campus-based open learning Background Web-based support for campus-based open learning The use of Web-based resources to support learners at a distance has been widely investigated in recent years, with many authors describing the benefits that it can bring. Such benefits include increased flexibility for learners (Nikolova and Collis, 1998), its ability to support discussion (Nixon and Salmon, 1996) and its wider benefit for the institutions involved in terms of student recruitment (Hanna, 1998). Whilst some of the benefits that apply to distance learning will also accrue for students based at traditional institutions, it would be naive to assume that all will apply, or that they will all result in the same patterns of student engagement. However, the particular features of this niche form of Web-based support remain ignored; even substantial investigations have grouped campus-based open learners alongside their distance counterparts as a contrast to students in traditional classes (for example, Hiltz, Coppola, Rotter, Turoff and Benunan-Fich, 2000). To date, much of the research on the use of Web-based support for campus-based courses has concentrated on the implementation of systems (for example, Shaikh and Macaulay, 2001), or has focused on the results of a particular case study. Few have attempted to draw out the wider implications of these for the design of Web-based support for open learning on campus, although there are some notable exceptions (such as Boyle and Cook, 2001). In part, such caution is entirely appropriate, given the problems of generalizing from case studies of educational impact (Oliver, 2000). However, one valuable way in which case studies can contribute to our general understanding of an area is by challenging received wisdom through the provision of counter-examples. There is thus a need to reconsider some of the benefits traditionally associated with online learning in the context of campus- based open learning. Use of the Web in dental education Computer-assisted learning has been used as a part of undergraduate teaching in dentistry, but is perhaps more often seen as a resource suitable for postgraduate training. Importantly, software packages (such as CD-Roms) still seem to be the dominant form of computer-assisted learning in this area, with current research in dental education still coming to terms with now-familiar issues in learning technology research such as the reluctance of students to lose contact with their tutors and the difficulties of moving beyond affective responses to consider actual changes to learning (see, for example, Lechner, Thomas, Bradshaw and Lechner, 2001; Welbury, Hobson, Stephenson and Jepson, 2001). However, Web-based courses are now being introduced for Continuing Professional Development (CPD) purposes - an area that is growing in importance for dental professionals (Grigg and Stephens, 1998). This is, perhaps, unsurprising given the pedagogic relevance of Web-based courses for learners in work (Bradley and Oliver, 2002). One reason for the relatively low level of uptake is that the value of such approaches has been called into question for certain key topics such as clinical decision-making skills (for example, Kay, Silkstone and Worthington, 2001). Consequently, many educators in this field remain in doubt about the viability of Web-based approaches to teaching. 52 ALT-J Volume 10 Number 2 The context for this study Against this backdrop, a decision was made to redevelop a programme in Dental Public Health for the M.Sc. of the University of London and the Diploma of the Royal College of Surgeons. Traditionally, this had been delivered through a one-year, full-time programme involving traditional face-to-face teaching methods. Cohorts included UK and overseas graduates; there are usually around equal numbers of each on the course. In recent years cohorts have been small, rarely growing larger than ten students. The prospect of a partnership with an overseas institution led to questions about how the course could be redeveloped to support distance students. A distance-learning version of the course was available, but was not Web-based; rather than relying on this, it was decided that the potential partnership presented an opportunity to explore whether a Web-based format would be appropriate for this course. This exploration was carried out by piloting a Web-based approach to teaching and learning on two modules of the course. The modules were delivered using WebCT, which is the institution's supported Virtual Learning Environment software. In order to ensure that the inferences drawn from this pilot were appropriate for the target group of students, the pilot was carried out by offering the course to students currently studying on the traditionally taught master's degree. As a result, the students who participated in the pilot used a Web-based course to support what was, for them, a campus-based education. The course that was developed drew on many of the standard features of WebCT, and the initial module made use of Web pages, bulletin boards, assessment facilities (multiple choice, answer matching, etc.), essay submission and administration facilities, reference management resources, and so on. The students on the course were unfamiliar with WebCT, and so an introductory page was developed (Figure 1) that introduced students to all of the features of the environment that they would need to use during the course. Students worked through this, with tutor support, as part of an initial face-to-face workshop, and the materials remained available from the course home page as a point of reference. Getting started with the Dental Public Health course What this document is fir This documentwelcomesyoutothe course, tells you about WebCT and contains some exercises to help you gain confidence with working in WebCT. It will probably take you 30-60 tninutes to work through it. Start by reading from the top, attempting the exercises as you go. Remember that this document has been opened in a new window. This means that you can swap between this document and WebCT easily, without having to close either of them. This should help you when you're trying to follow the instructions for the exercises. .__ You have now arrived at the very first module of our course in Dental Public Health! Our course has been developed jointly by the Eastman Dental Institute and Education and Professional Development of University College, London and represents a new venture in the subject. For the Figure I: An excerpt from the introductory materials for the course " S3 Ruth Holt Martin Oliver and Claire McAvinio Using Web-based support for campus-based open learning In addition to the materials and resources provided for the students, collaboration and discussion was encouraged. Structured discussion areas were set up, where students were asked to post answers to questions, to discuss other students' responses, and were encouraged to ask questions of the tutor and each other. Several assessment features were used, including formative self-assessment and assignment management. From an early stage in the design process, it became apparent that the use of WebCT could support pedagogic approaches that were novel for this particular course, such as the introduction of interactive assignments, which involved making judgements about images (for example, Figure 2) and analysing the resulting data using spreadsheets. These assignments closely resemble authentic problems facing practitioners. Look at the first two sets of bitewing films and the completed example spreadsheet to make sure that you understand exactly what to do and how to enter your scores on the worksheet. (To look at the example spreadsheet, go to Excel, and click on the tab at the bottom of the page that says, "example". This contains a worked example for the two set of bitewing films shown in the table below.) You may find it helpful to print this page off so that it is easier to compare the images with the numbers in the spreadsheet. Figure 2: An image and excerpt from the assessment on judging bitewing radiographs Another important development was the introduction of self-assessment activities in the second module. This reflects wider concerns within the field about practitioners' ability to assess their own ability, and the relevance of training students in self-assessment skills as a vital component of CPD (see, for example, Evans, 2001). The students developed criteria for self-assessment at a second face-to-face session, based on their analysis of the strengths and weaknesses of a pre-prepared 'bad practice' document. Students were then asked to apply these criteria to their own work, in order to assess their own strengths and weaknesses and to grade their performance. Case study methodology To evaluate the pilot, several sources of data were considered. These included direct observation of students, monitoring their use of resources and participation in discussions using WebCTs student tracking facilities, and, most importantly, a focus group and interview. This range reflected both the small number of students on the course (three, together with additional participants who volunteered to be involved but were not formally assessed) and the emphasis on understanding students' experiences through considering both their accounts and our perceptions of the course. These data were analysed following the procedures described by Kvale (1996) for ad hoc meaning generation. — An-] Volume 10 Number 2 Results and discussion Four main issues arose from this study: assessment, experiences of self-assessment, the role of flexibility and online discussion. Each of these will be considered in turn, and will be illustrated by quotes from the focus group and interview with the students. Assessment issues It was clear from tracking usage of WebCT, and from comments in the focus group, that the students were assessment-driven. This is hardly a surprising finding, echoing as it does other work on assessment (see, for example, the discussion of 'backwash' from assessment in Biggs, 1999). However, the small cohort permitted a detailed level of observation and discussion that allowed the implications of this to become clear, revealing some important features of the students' engagement with the course. The direct implication of the influence of assessment was that non-assessed activities were neglected, and time was allocated to work in proportion to its perceived status as contributing towards a final score. With [this activity] having formal assessment assignments, deadlines and marking, you will almost instinctively give it a higher priority than you will other areas, because that is the way we have been brought up and educated. Although they were assessment-driven, they did not particularly like being assessed. However, they saw it as being vital to their learning process - not because of the feedback that they received (although they did recognize the value of this) but simply because it forced them to engage with the material they otherwise used superficially. I think, to be frank, who likes to do the assignment? Who likes to do the deadline? You have to submit this assignment by the end of December, January. In the learning process, you have to do . . . you have to do the assignment, and by then only you learn, you learn something. So I think the assignment is very beneficial, because it makes you t o . . . it pushes you to do something, rather than [the alternative where] you are just left to do your own reading. Because I think you won't do [the reading] until the end of course when the exam comes. This lack of engagement was systematic; it was not merely a symptom of this particular course. For the students, their various lecturers' use of assessment represented an important use of power that coerced them into committing time and effort to particular topics. I can get away with not doing the background reading for lecture X because I'm not likely to be found out. I can cover it. But if the assignment's not in by the computer time, then it's just black and white, isn't it. It's either there or it's not there. I think if you're going to put in something that's not covered anywhere else, then just by the structure of this, or the weighting of the grading of the other areas, you will ensure that we do that. Although students discussed the competing demands of different parallel modules on their time, they made it clear that such coercion led primarily to the sacrifice of leisure time. I think the answer for us is, we like assignments because it keeps us thinking, or it makes us think about this course rather than going down the pub tonight. — Ruth Hoh. Martin Oliver and Claire McAvinia Using Web-based support for campus-based open learning These responses seem to paint a cynical, calculating picture of the student experience. However, drawing on Bourdieu's notion of different forms of capital (1986), they can be interpreted instead as representing a rational expenditure of precious personal capital - their time - in order to gain the maximum symbolic capital (qualifications and grades) in return for their investment. Students' experience of self-assessment Further assessment-related issues arose from the self-assessment exercise described above, which the students found to be extremely difficult. The creation of criteria for assessment was the first indication of the students' difficulties. As a group, the students were able to generate and agree criteria against which their work should be judged. However, they were not able to agree on the relative weightings each of these should be given. The weightings they chose for each group of criteria (layout, quality of analysis and use of evidence) appeared to reflect their understanding of the purpose of assessment, and so will be outlined briefly, under the headings suggested by the students as a rationale for their allocations. Rationale for allocation of marks Layout Quality Use of evidence 'Substance not spin' 24 38 38 Volume/length - rewarding effort' 35 35 30 'Masochism as motivation' 40 30 30 Table I: Students' percentage mark weightings for the three categories of criteria The first student weighted the marking scheme heavily towards the thoroughness and validity of the analysis, attributing relatively low importance to the structure and presentation of the documentation. (The weight he was willing to attach to this category arose almost exclusively from the need for a well-structured document.) This student defended his allocation on the basis that, if this were a real report for use in dental public health, the reliability of decisions based upon it would depend solely on the quality of the analysis. The second student spread the marks evenly across the three areas. She argued that this was fairer than over-emphasizing any one area, since a clever student who made no effort would score highly if analysis was allocated most of the marks. She explained that a recognition of effort, rather than an arbitrary measure of quality, was the most appropriate way to reward students. Implicit in this is a positive motivational model of the role of assessment, encouraging those who try hard to greater lengths. The third student adopted a different scheme again, placing the greatest emphasis on layout and then evenly dividing the remaining marks. This would force him to take this aspect of his work - which he knew that he was weak at - more seriously, he explained. He described this as a 'punishment' and even a 'masochistic' model of assessment, intended as a deterrent to laziness. After discussing these different models, the third student agreed that his mark scheme was too personal to be appropriate for everyone in the group. (This illustrates another important distinction between the three; he viewed an ideal assessment scheme as being 56 ALT-J Volume 10 Number 2 tailored to the individual being assessed, whilst the second believed that the allocation of marks should be tailored to the students' ability and the first felt that marks represented an objective set of criteria.) He was willing to adopt the mark scheme of the second student, but not the essentially positivist scheme of the first student. Similarly, the first student refused to accept the appropriateness of the other two mark schemes on grounds of validity. Consequently, it was agreed that each student should use all three mark schemes to grade their work, as would the course tutor, and that marks would be compared both across schemes and across markers in order to enquire into the ways in which these different value systems rewarded the same piece of work. Eventually, however, only one of the students completed this process. The first student, refusing to accept that he (or any other non-expert) could validly mark the assignment, opted out of the exercise completely. The second student prepared a qualitative assessment of her work (which the tutor agreed was insightful and appropriate) but said that she felt it was inappropriate to award herself marks using any of the schemes. Although the third student completed the exercise, he said that he did not take the marks he had given himself par- ticularly seriously. Interestingly, in spite of difficulties of grading, both the second and third students felt that the qualitative element of the self-assessment had been extremely valuable. Clearly, the problem that arose in this case was not one of understanding or ability, but was instead a cultural issue. Because this form of assessment places the responsibility for devising marking criteria and judging work with the student, it represents a radical challenge to traditional forms of power and hierarchy in education (Rowland, 1993: 136-40). Such a challenge left the students feeling uneasy. Although they were all capable of completing the process, they could not accept it as an appropriate part of their course given their previous experiences of education. In effect, engagement in such self-evaluative activity simply was not part of the students' habitus (Bourdieu, 1977). This suggests that attempts to introduce techniques such as self-assessment will need to take careful account of students' educational histories. It may be necessary to build up to the introduction of such an unfamiliar technique using less radical methods first in order to bridge the cultural gap between past experience and the desired form of learning. Flexibility As outlined above, the importance of flexibility in the literature suggests that it would have a beneficial effect for learners. The course described here did make studying more flexible; for example, students used WebCT to access course materials from home, and from other sites, and at a range of times outside scheduled teaching hours. This helped balance the workload for the module against other course commitments. However, flexibility was not a straightforward virtue. Firstly, although students valued the potential of WebCT'to support flexible study, they did not necessarily use it in this way. From my point of view, it's got to be [flexible] - that's a big bonus. Particularly if you're travelling. But on a practical level I've only accessed it when I'm in here. I mean, it still allows me some flexibility, I can do it when I've got a moment. I really should be doing it, for the sake of coming in. In fact, being able to do work 'when I've got a moment' was a problem for two of the three students, who would have preferred scheduled sessions: 57 Ruth Holt, Martin Oliver and Claire McAvinia Using Web-based support for campus-based open learning 1: I think if we have a proper session for the WebCT, if we have [this], then we have, we can . . . [pauses] If we can do it whenever we are free, then some . . . we keep on doing something else rather than doing WebCT. So if we have a proper class for WebCT I think it's a bit better. 2: Yes, it's better at pushing you, you know. Do it, do it! 1: Yeah. You have to do it. [laughs] With all three students, increasingly flexibility led to work being put off, which they acknowledged would cause problems in the longer term. Assessment was seen as being particularly important as a way of counteracting this tendency. I personally prefer to be kicked up the backside to do something continually because then I know by the end I'll have done it. Whereas most of us, out of human nature, would say, I don't have to do that now. And then you get into a mad panic come June or whatever. Even with assignments, however, observation and discussion with the students showed that no matter how long the lead-time the work was completed as close to the deadline as possible. What these experiences illustrate is that there is a need for a balance between flexibility and structure in courses, that this may be difficult to achieve, and that it will vary according to the needs of individual students. Online discussions In spite of drawing on a range of guidelines for practice (such as Salmon, 2000), online discussions did not work well. The students felt that the structured format reduced the exchanges to a transmission of information. You know, there isn't enough discussion that goes on. There's too much . . . sometimes, you know, we might as well just have a handout. The students did acknowledge that this was, at least in part, a consequence of their own choices. I mean, the opportunity to discuss is there, but we as students don't always take up that challenge perhaps. In part, they viewed this as a consequence of the fact that their contributions to the discussion board were unassessed, and thus were viewed as being optional. That's why you see our postings up for these things is much poorer than the assignments. If they are not graded. Yes, they are important, it's just that a lot of important points there, and you have to follow it up, but it's not graded, so . . . [pause; laughs]. That's the point of view, you know. You just put it as something for when you have time, and you want to have some fun, you have the mood to do it. Another reason for this choice was that they felt that online discussion was inferior to face- to-face exchanges: I don't think it's the best way to discuss matters, I prefer to do it around a table. You see the answer that much quicker coming back, you get more from it, and if you don't 58 ALT-J Volume 10 Number 2 understand something you can ask directly. That's my point. I don't see that as a strength of WebCT. I don't use it for discussion. I just don't. I use it for the learning tools, like the assignments. If I want to discuss, then I'll do it in here, because you get so much more out of it. It's a barrier for me to discuss something like that. Given that this was a hybrid course it was simply easier and better to discuss issues when they met. However, they agreed that such discussion was better than nothing. I think that if it's the only option you've got for discussion then it is better than none at all, but as a replacement for sitting around a table, I think it's got a long way to go. I described it as a barrier because it's a barrier as opposed to just doing this sort of conversation. But it's a facilitator if there's no other way of conversing, isn't it. It just depends on what standpoint you're looking at it, and I'm absolutely, you know, sure that given the alternative between only having a once a month visit to your tutor and having the opportunity to communicate through a fairly instantaneous medium then that is a step in the right direction. But I don't think it replaces sitting round and discussing it in a group. Importantly, this reaction against the online discussions seemed to represent an exercise of power by one of the students. Of the three, the second and third (as described in the section on self-assessment) had contributed sporadically to the online discussions. In particular, the second student had posted several long, thoughtful messages. This contrasted sharply with her more hesitant, infrequent contributions in the face-to-face discussion. She explained that this was because she was able to reflect before writing, and was able carefully to compose and revise messages before sending them. This was something that she felt to be of great value given that English was not her first language. By way of contrast, the first student had avoided the online discussions wherever possible. He dominated traditional group discussions, partly because of his quick wit, but also because he was more fluent and articulate than the other two students. (He was the only one of the three for whom English was the first language.) He identified his dyslexia (which he had previously kept hidden from the group and the tutor) as one particular reason for preferring to speak than to write. Thus the face-to-face discussions reinforced his position of control in the group, whereas the online discussions allowed the other two students - and particularly student two - to overshadow him. It would be naive to argue, on the basis of this case, that moving discussions online will thus eradicate or even diminish power structures within groups. However, what this study vividly illustrates is that such a move will change these structures in ways that are influenced by personality, background and fluency in writing and speech. As a result, these characteristics are also likely to inform students' preferences for (and the potential success of) a particular medium for discussion. Other issues In addition to these main themes, several other points were noted. Firstly, although students found some elements of WebCT frustrating, overall they were satisfied with their use of it. 59 Ruth Holt Martin Oliver and Gaire McAvinia Using Web-based support for campus-based open learning It's not a problem - it's just that when you don't know something, it's so annoying that it takes so long to work it o u t . . . You end up swearing at the screen, don't you, for some little thing like that. I think it's a pretty good sign for the system if between the three of us the only major concern is a little glitch like the type of file that has to be submitted. I mean, it does suggest that it's simple to use. Moreover, they valued the novelty of this approach. I think that WebCTis another way to add to the variety of the learning process. Methodologically, the focus group also highlighted the fact that logs of students' use of the system gives a flawed and partial account of their experience of the course, echoing the warnings of Jones (1998). I tend to spend most of the time on the assignments. I tend to print the content anyway, so I've got a record of it. Conclusions At a superficial level, the study demonstrated the success of using WebCT to deliver courses of this type. The students all completed the course, and were generally well disposed towards the use of the Web to support their learning. However, closer examination shows that the course design, which had been carefully based on recommendations from the literature, caused a number of problems. What this study illustrates is that not all students are equally well suited to learning from Web-based materials. Thus the success of any such approach will be influenced by students' personalities and educational history as much as by the design of the course. The wider implication of this is that courses should, ideally, reflect the characteristics of the individual students on the course. The students described here (two non-native English speakers, and one dyslexic student) were not 'typical' - but this is the point. No student is 'typical', even if their distinctiveness is not initially obvious (as was the case here with the dyslexic student); each will bring with them personal histories and dispositions that shape their interaction with the course, and will thus determine its effectiveness. In effect, then, the idea of 'student-centred learning' should be moved from being simply a set of pedagogic strategies applied to all students, to being an integral part of course design during which the suitability of all teaching methods, including those not traditionally considered to be student-centred, should be considered. Perhaps more insidiously, this conclusion also implies that, given the increasing diversity of students (even within the context of specific courses), it may be impossible to create any single course that is well designed for each and every student in a cohort. Also linked to the issue of individual educational history were the problems associated with the introduction of self-assessment. This was entirely new for the students on this course, and thus whilst it had the potential to provide considerable benefits, its success was limited because it was unfamiliar and challenged deeply held assumptions about the way education 'ought' to operate. 60 ALT-] Volume 10 Number 2 Flexibility, which is almost ubiquitously claimed as an advantage for courses involving the Web, proved to be a mixed blessing. Although students spoke of the value of studying when they liked, they also admitted that the reality was that they typically accessed materials on campus, and organized their work around deadlines. Moreover, they indicated that more, not less, structure to the course would have helped them use their time more effectively. Thus the potential of the Web to support 'any time, any place' learning became not simply irrelevant but also detrimental to the students' management of their learning. This would imply that the current advocacy for flexible courses may have gone too far in its attempts to address the dilemma of structure identified by Rowland (1993:45-6). The study also echoed other research which has emphasized the centrality of assessment in students' perceptions of courses. Given that students organized their time around dead- lines, the number and pacing of assignments will create the structure of the course, at least in terms of student work. Inherent in all of these issues is one common problem: students' commitment of time to a course comes at the expense of time which might otherwise be spent elsewhere. This observation allows the often-criticized phenomenon of student instrumentalism to be reframed not as laziness or superficiality, but as a shrewd way of investing their personal resources (Bourdieu, 1986). It is only rational to avoid making any more effort than necessary, by opting for the most familiar, convenient ways of completing the course. The short-term planning witnessed here illustrates that a lack of clear course structure can add to, rather than diminish, this tendency to avoid committing time to the course, at least for those students who do not find study intrinsically motivating. This has significant implications for the kind of hybrid course described here. Where a Web-based course is used to support campus-based education, students may well drop elements that they find difficult in favour of easier, more familiar options. Unless they feel an inherent motivation to explore and experience these new teaching methods, it will be necessary to ensure that the innovative elements are either mandatory or easier for all the students involved. References Biggs, J. (1999), Teaching for Quality Learning at University, Buckingham: Open University Press. Bourdieu, P. (1977), Outline of a Theory of Practice (2001 edition; translated by R. Nice), Cambridge: Cambridge University Press. Bourdieu, P. (1986), 'The forms of capital', in J. G. Richardson (ed.), Handbook of Theory and Research for the Sociology of Education, New York: Greenwood Press, 241-58. Boyle, T., and Cook, J. (2001), 'Online interactivity: best practice based on two case studies', ALT-J, 9, 1, 94-102. Bradley, C. and Oliver, M. (2002), 'The evolution of pedagogic models for work-based learning within a virtual university', Computers and Education, 38, 37-52. Evans, A. (2001), 'Assessing competence in surgical dentistry', British Dental Journal, 190 (7), 343-6. 61 Rutfi Holt Martin Oliver and Claire McAvinia Using Web-based support for campus-based open learning Grigg, P. and Stephens, C. (1998), 'Computer-assisted learning in dentistry: a view from the UK', Journal of Dentistry, 26, 387-95. Hanna, D. (1998), 'Higher education in an era of digital competition: emerging organizational models', Journal of Asynchronous Learning Networks, 2 (1), 66-95, http://www.aln.org/alnweb/journal/vol2_issue1/hanna.htm, URL last visited 5 April 2002. Hiltz, S., Coppola, N., Rotter, N., Turoff, M. and Benbunan-Fich, R. (2000), 'Measuring the importance of collaborative learning for the effectiveness of ALN: a multi-measure, multi-method approach', Journal of Asynchronous Learning Networks, 4 (2), http.//www. aln.org/alnweb/journal/Vol4_issue2/lelhiltz/le-hiltz.htm, URL last visited 5 April 2002. Jones, C. (1998), 'Evaluating a collaborative online learning environment', Active Learning, 9, 31-5. Kay, E., Silkstone, B. and Worthington, H. (2001), 'Evaluation of computer aided learning in developing clinical decision-making skills', British Dental Journal, 190, 554-7. Kvale, S. (1996), Interviews: An Introduction to Qualitative Research Interviewing, London: Sage. Lechner, S., Thomas, G., Bradshaw, M. and Lechner, K. (2001), 'Planning oral rehabilitation: case-based computer assisted learning in clinical dentistry', British Dental Journal, 191 (3), 152-6. Nikolova, I. and Collis, B. (1998), 'Flexible learning and design of instruction', British Journal of Educational Technology, 29 (1), 59-72. Nixon, T. and Salmon, G. (1996), 'Computer-mediated learning and its potential', in R. Mills and A. Tait (eds), Supporting the Learner in Open and Distance Learning, London: Pitman Publishing. Oliver, M. (2000), 'Evaluating online teaching and learning', Information Services and Use, 20, 83-94. Rowland, S. (1993), The Enquiring Tutor Exploring the Process of Professional Learning, London: Falmer Press. Salmon, G. (2000), E-moderating: The Key to Teaching and Learning Online, London: Kogan Page. Shaikh, A. and Macaulay, L. (2001), 'Integrating groupware technology into the learning environment', ALT-J, 9 (2), 47-63. Welbury, R., Hobson, R., Stephenson, J. and Jepson, N. (2001), 'Evaluation of a computer-assisted learning programme on the oro-facial signs of child physical abuse (non-accidental injury) by general dental practitioners', British Dental Journal, 190 (12), 668-70. 62