mason Australian Journal of Educational Technology 2000, 16(2), 173-200 Factors affecting students’ satisfaction on a web course Robin Mason and Martin Weller The UK Open University This paper takes the form of a dialogue between the evaluator and the course team chair of a very large web-based course presented by the Technology Faculty of the UK Open University. An extensive evaluation has been conducted following the first pilot presentation of the course, and the two authors discuss the findings as they relate to students’ satisfaction with the course. Seven key issues are raised: skills versus academic content, students’ previous computing experience, interaction through computer conferencing, online group work, online tutoring, students’ lack of time, and revising a course in the light of evaluations. Finally, the results of this course are compared to three other web courses. Introduction This paper takes the form of a dialogue between the two authors, one the evaluator and the other the course team chair of an innovative web-based course at the UK Open University (UKOU). The evaluator has collected extensive feedback from students and tutors on the first presentation of the course in 1999, through web questionnaires on each module of the course, through student activity logs and through telephone interviews. The course team chair authored a third of the course and led the team of academics, editors and web designers who developed the course. He also supported tutors and students during the first presentation and responded to several hundred emails. The UKOU is an adult distance teaching university which has built its international reputation on the quality of its printed course materials and its supported open learning approach to course delivery. Its technology strategy is to use new media as they enhance this process, increase access for students and add value to the teaching and learning experience. Telecommunications technologies, especially text-based asynchronous 174 Australian Journal of Educational Technology, 2000, 16(2) interaction, has been implemented widely across many disciplines, as has multimedia in the form of CD-ROMs. Real time Internet and web technologies are beginning to be useable, now that home based access is available to significant proportions of the 250,000 student population of the University. The discussion which follows draws out the complex web of reactions students have to the ‘technologisation’ of higher education, and explains one approach to this complexity through a particular, innovatory course. The course was T171 You, your computer and the Net. It is an entry-level course about information and communication technology (ICT), delivered entirely over the web with online tuition. The course is studied part-time over 32 weeks, and requires about 200 study hours. The course consists of three modules: • Becoming a confident computer user – an introduction to basic computer skills and applications, using the Internet and group working. The material was taught in a generic manner, so it was not software package specific. • The story of the personal computer – using the set text, Accidental Empires, (Cringely 1996) to tell the basic story of the development of the computer, the module explores technological, social and economic issues raised by the material. • The story of the Internet – using the set text, Where Wizards Stay Up Late, (Hafner and Lyon 1998) again to tell the basic story of the development of the Internet, the web site material explores issues such as the development of protocols, paradigm shifts and social impact of the Internet. There are four tutor marked assignments (TMAs) during the course and an end of course assessment (ECA) replacing the conventional exam. Students have to construct their work as HTML documents and submit them electronically. The course has been described in detail elsewhere (Weller 1999). The course was piloted in 1999 with 850 students. 1. Teaching skills or an academic discipline? One of the key issues which emerges from the student feedback data is the dissatisfaction many students (nearly a third) felt about the lack of skills training they received on the course: Mason and Weller 175 “I wanted more experience on using search engines.” “More web creation skills eg. tables, frames, forms, scripts.” “More hands-on and less reading and detail about the development of the Internet.” “I’m not interested in how the Internet started; I want a job and need practical experience.” The course purports to have students creating web pages, word processing and generally managing their personal computer by the end of the three modules. Yet the second two modules of the course are largely devoted to the history of the Internet and of the web. A good two thirds of the 150 or so students who filled in questionnaires found this balance between the ‘how’ and the ‘what’ very appropriate (Chart 1), and appreciated that the background to the Internet and the web helped them build a solid understanding on which to base the particular skills of web design etc: “If you don’t know the development, you don’t understand the issues of today.” “Modules two and three covered a broad range of areas. They opened up many channels which I could investigate further.” Chart 1: Percentage of students reporting an increase in understanding This course is about the personal computer and the Internet and what these technologies can do. It is obviously appropriate, in fact necessary, that it teach skills in searching, managing, browsing and creating information. But these skills are not the traditional stuff of an 176 Australian Journal of Educational Technology, 2000, 16(2) undergraduate degree; usually in pre-computer days, one assumed these information handling skills would be picked up along the way. So on the one hand, there is the tacit academic view that we don’t give credit for skills but only for understanding; on the other, there is demand from students for job-related skills. How do you see this conflict as a course designer? Do you think that the digitisation of information has changed the balance between skills and understanding? How do you respond to the many students of this course who feel angry, misled or disappointed about the level of their skills at the end of the course? Course Chair's response This was one of the issues which concerned us greatly in the course design stage. This tension between training and academic level arises in many ICT courses, for instance learning a programming language and learning the principles of software engineering. As an entry-level course which can count toward a university degree, T171 needed to be of a suitable academic standard. We were aware however that many students wanted to learn software skills, and that many were interested in this one course, and had no intention of studying further. Module one of the course has an activity based approach, and introduces students to common applications such as word processors, spreadsheets, graphics etc. It also gets students online, using e-mail, and writing web pages. At one level this can be seen as training, but it is integrated with academic material about group formation, communication, clients and servers, and so on. There is also a strong emphasis on learning to learn, so students new or returning to study can develop their study skills. With the remaining two modules, one of the aims of the course team was to bring students into the ICT culture. A powerful tool for enculturation is the use of narrative, which is why the modules make use of historical accounts of the development of the personal computer and the Internet respectively. However, the modules are not really about the history of these technologies, but rather they use these narratives to provide a structure from which to cover a broad range of areas, including the technology in detail, the impact of these technologies upon society, the nature of technological development and the ICT industry. In many ways we are tracking a moving target with regard to the skills our students need. The level of computer skills is constantly improving, so what a course needs to include one year may become assumed in later Mason and Weller 177 years. This is particularly true of a level 1 course delivered via this medium. Many students were completely new to study, and to the medium itself, and some account must be taken of this. People are very familiar with print and text, but not so with web sites. Thus in some respects we are introducing people to a new learning medium. Again one of the reasons we chose a narrative based approach was because of the familiarity and comfort it offered in this new medium. Other research has shown that narrative can provide structure for multi-media learning material (eg. Laurillard 1998). Many students feel they want the skills training because this will help them with employment. However, there may be a mistaken belief about what employers are looking for. It is increasingly the case that employers are looking for more than just skills in one area, but rather transferable skills, or at least evidence of a broader knowledge. I think T171 gets the balance about right here, in that it provides sufficient skills, such as basic HTML, and it gives students the context and opportunity to develop these further, for example through quite open assignment specifications. By giving students the confidence to use software and a meaningful context in which to develop it, their software skills can go beyond that which is taught explicitly in the course, and indeed beyond what they would gain from a straightforward training course. This was borne out by the sophisticated web sites many students created for their end of course assessment. Students also gain an appreciation of the wider implications and possibilities of these technologies. Since the 1999 presentation finished several students have reported that the course has been significant in gaining them new employment: “I thought you would all like to share in my good news. I applied for a job with [deleted] and had my Board last Friday. I've now been offered a training attachment with them. They were very impressed with how much I knew about the internet and web pages ... if you're looking for work in this sector - start applying - we certainly seem to have grasped what employers want to hear! “ As with many courses, students’ expectations about the course play a critical role in their ultimate satisfaction. For the 2000 presentation we have produced some preparatory material, and this states very clearly that students should consider T171 an academic course, and not a training course. So, I feel we provide enough of a training element so students can gain new skills, without compromising the academic nature of the course. As long as students are made aware from the outset of the nature of the course, then I think they will be satisfied with what it delivers. 178 Australian Journal of Educational Technology, 2000, 16(2) 2. Students’ previous computing experience This course is a first year undergraduate course in the Technology Faculty. It requires no prerequisite knowledge and no previous experience with the computer. Yet amongst the first cohort of students there was a very wide range of abilities (Chart 2): from those with absolutely no computer experience right through to experts in various aspects of computer use. Chart 2: Students starting knowledge The pervasive nature of the computer has led to a very wide range of abilities, attitudes and experiences of its use: for example, one student had built several computers as a hobby, but had absolutely no experience of the networked computer and had never accessed the web. Another student taught databases at his university, but he said he had real difficulty with the technical side of this course. Many students who were entirely new to computers went through a very steep learning curve especially at the beginning of the course. Some dropped out complaining that it was far too difficult; most of those who stuck with it until the end reported a tremendous sense of achievement: “I nearly gave up several times and I was absolutely knackered most of the time keeping up with my job and the course, but I am very satisfied that I met the challenge and stuck it out.” “It was an incredible struggle, but I enjoyed it.” Mason and Weller 179 Preparing a course about the technology for people with such diverse backgrounds must be an almost impossible task. How can you expect to keep the experienced students challenged and yet provide the high levels of support and introductory level of materials for complete beginners all on the same course? Amazingly, by far the majority of students on the first presentation were very satisfied with the course (Chart 3)– how do you account for this? What do you have to say to those students who complained that the course did not cater for new comers to the personal computer? Chart 3: Student satisfaction levels by module Course Chair's response We deliberately set out to create a course with broad appeal, both in terms of computing experience and subject area. Given the pervasiveness of ICT the course was aimed at students from all areas, not just technology or computer science. In 1999 70% of the students were new to the UKOU. However, one of the problems was that in the pilot year the course mailing did not go out sufficiently early. This meant that computer novices did not have sufficient time to get comfortable using their PC before the course started. For the 2000 presentation we have produced a preparatory activities booklet, which contains optional activities designed to help newcomers become accustomed to using the Windows environment, navigating around a web site, and thinking about study skills. This was mailed in November 1999, with a course start date of February 2000 and so should allow the novices to feel more comfortable when the course starts. 180 Australian Journal of Educational Technology, 2000, 16(2) At the other end of the continuum is the problem of keeping happy those students who already have a good grounding in this area. We tried to achieve this through several means. The first of these was the use of group work via computer mediated conferencing (CMC), which many students had not experienced before. The more technically advanced students often found it interesting and rewarding to help their fellow students via this medium at the start of the course. Another way in which we tried to keep the course interesting was to cover a broad range of material, as I mentioned above. So, for instance someone with a technical background may have thought relatively little about the social impact of the technologies, or the management structures of ICT companies. Here students with little computer experience but with broader social or management backgrounds may be able to provide input. In addition, the assessment, particularly in modules two and three is quite open-ended. Students choose from one of two titles, and are asked to produce a ‘web-essay’. Students with a good grounding in the material already can use the web as a research tool, and integrate images and links into their document. Many students found the assessment a rewarding task to perform. People have many reasons for taking courses, sometimes it is to gain knowledge about a totally new subject, to gain accreditation for knowledge they already have, to work towards a specific degree, or simply to consolidate knowledge they have gained on a ‘piecemeal’ basis over the years. This is borne out by comments such as: “ I am enjoying the course enormously. I have found it very difficult at times, frustrating at times, and very challenging, but I have managed to keep going! My previous experience has meant that I had lots of ‘bits’ of knowledge but no coherent overall understanding. The course is beautifully constructed and I can really appreciate just how much work has gone into making it work, having prepared courses myself (at lower level). It is a delight to be a student and to be so expertly guided through the course!” 3. Computer conferencing: Underwhelming or overloading? There has been a good deal of evaluation of the use of computer conferencing as a means of interaction between students and tutors on distance education programmes over the last ten years. Indeed the UKOU has produced a good proportion of it (eg. Mason, 1998; Salmon, 1999; Wegerif, 1998). One way of characterising its use as an educational medium, is to say that its strengths are also its weaknesses: Mason and Weller 181 • it doesn’t require fixed times for study, but consequently other demands on one’s time easily take precedence • it maintains a record of all interactions - but this makes many people wary of committing their ideas to such a public forum • it allows everyone to be ‘heard’, but this leads to an overload of messages which many find completely overwhelming. One of the paradoxes of this medium which is very apparent in the student feedback of this course lies in the disparate perceptions that on the one hand, there were too many messages or that, on the other, there was too little participation. Nevertheless, as is usual with computer conferencing, there were many students who found the medium very satisfying: “I couldn’t have coped without the conferencing.” “The subject conferences are the best part of the course.” “Because of the mix of people, there is a lot of self-help.” “I wouldn’t have understood nearly so much of the course if there hadn’t been the support conferences.” However, the dissatisfactions expressed about computer conferencing on the course were contradictory. Newcomers to the computer tended to find the number of messages overwhelming and the competence and tone of the messages from experienced computer users very off-putting. At the same time, one of the biggest complaints about the tutor group conferences was the lack of participation: “I felt intimidated by the level of knowledge some people displayed in the conferences and by the manner in which they did it.” “We need more small group conferences and people should be required to participate.” “I found the online conferencing unusable. I tried a few times to get into the discussions, but without success. I would have had to be logging in every day for it to work and this was impossible.” “The most disappointing thing about the course was the lack of participation in the conferencing.” “Most people seem to be too busy to contribute to the conferences and this is a real pity.” The course conferencing environment consisted of tutor group conferences in which each tutor and the 12-15 students assigned to every tutor were expected to raise course related issues and problems. In addition there were course-wide subject conferences run by central academic staff to 182 Australian Journal of Educational Technology, 2000, 16(2) discuss the major topics in each module. Finally there were technical support conferences on topics such as word processing, networking, databases etc. These were moderated by specialist staff. Some of the conferences worked better than others (Chart 4) and what some students regarded as “life-saving”, others found off-putting. A number of students suggested that those new to computing have their own groups and conferences, whereas many students found the support of the experienced students invaluable. How do you cater for this range of requests? Can you satisfy the conflicting demands of students in this unpredictable medium? How do you regard the lack of online participation? Chart 4: How well are the support conferences working? Course Chair's response Given the aim of T171 to introduce students to computers and the Internet, it would be difficult to conceive of the course not having a strong conferencing element. We saw the use of conferencing as a vital means of engaging students, as well as the means through which they could gain support. Sending and receiving messages was thus the first activity in module one. Many students did not contribute to the national conferences, but they did read messages, and this can be very useful. As I mentioned above, for many people this is a new medium, and they are learning the appropriate Mason and Weller 183 skills. Some students reported that they felt they had to read all messages in every conference, and consequently spent a lot of time reading irrelevant messages. One student commented: “I wasted a lot of time reading messages which were irrelevant (to me) and in the end had to push and rush to get my TMA done. I do understand that this is completely my own fault but I find reading messages addictive and just have to look at the next one!” As many students are new to study, they felt frightened of missing something so read every message. As the course progressed we tried to give students appropriate analogies for using the discussion conferences, for instance saying they were more like a party where you could join a conversation if you wished. This was to try and encourage students to be selective as to which discussion threads they followed. However, although we will continue to stress in our materials that students are not expected to read all of the messages or even all of the conferences, it may be that some students can only develop the confidence to do this through experience. Active discussion has long been one of the aspects which is difficult to provide in distance education, with tutorials and summer schools being the usual means of achieving this. In T171 we deliberately wanted to provoke discussion on a range of issues, so ‘embedded conferences’ were placed within the web based course material. Here students were encouraged to go to the conference and discuss issues such as the role of Microsoft in the industry. As well as providing further interest to the material this helps students test their understanding of concepts through dialogue, which can then be refined iteratively. This learning model of conceptualisation, construction and dialogue has been successfully applied on other OU courses (eg. Weller and Hopgood 1997). However, many UKOU students choose to study at a distance precisely because they prefer to work alone, so there is an issue as to what degree the course should force people to participate. As UKOU students are adults we preferred to give them the opportunity to do as much, or as little, conferencing as they wanted or needed. We will continue to monitor the conferences, and encourage good moderator practice, so conferences do not become overwhelming for students. The use of conferencing must remain a vital element of the course, but we can do more to help students learn how to fit conferencing within their overall work commitments. 184 Australian Journal of Educational Technology, 2000, 16(2) 4. Online group work: A substitute for lack of face-to-face meetings? One of the central features of the traditional UKOU student support system is the face-to-face tutorial held in study centres around the country. This course is the first level one undergraduate course to dispense completely with face-to-face meetings. It does, however, include a number of group activities and an assignment requiring a group web page. There is a whole research literature on group work as a method of teaching and learning (eg. Tiberius, 1999 ) and a growing body of studies into its application in the online environment (McConnell, 1994; Hodgson and McConnell, 1995). Quite predictably, there were many complaints about the group work demands on this course: “It seems to have been forgotten that we students have never met, probably never would have met, and but for this course wouldn’t speak to each other and then somebody expects us to work together like a well- oiled machine, I feel your expectations were too high.” “There was a sharing of ideas, but little agreement or compromise. I found group working more of a hindrance to progress than a help.” “The problems are all practical. I travel a lot and have to do my coursework in concentrated bursts. I feel I’m letting the other members of the group down – catching up on the conferences is a big investment of time for small benefit.” “Trying to learn to communicate online was enough to manage initially; the necessity for group work so early on in the course created unnecessary pressure.” “Forget group work – People either do or they don’t. People (ie. British people) do not like to be told to do anything!” These kinds of responses have been noted on other OU courses with a group work component (Mason, 1995; Thorpe, 1998). Nevertheless, feedback from modules two and three questionnaires contained many complaints from students that there was no group work! Chart 5 gives a quantitative view, but comments included the following: “There should have been exercises involving group work as in module one. During module two, we drifted apart, just as we were grasping the concept of team working online.” “Module two was less engaging than module one – it was all reading and note taking, and no group activities.” “Needs more active encouragement for group conferencing to continue.” “I would have like more opportunity to work as a group; I felt rather isolated during module three as my group didn’t communicate much.” Mason and Weller 185 Chart 5: How well is your tutor group working? The course obviously aims to give students a taste of group working as one of the important features of networked computing. In other words, the course content justifies, even demands it. Chart 6 shows students’ response to this ‘demand’ in relation to the main national conferences. But does group work substitute for face-to-face meetings? Teachers who use group working practices in their campus-based courses would probably advise against trying to use it online, and especially not in mass distance education. Having seen the reactions of students for and against on the first presentation of this course, what are your thoughts about the design of successful online group activities? Course Chair's Response The group work aspect fulfilled a number of roles on the course. Firstly, by getting students to engage in an activity using CMC in module one, it encourages them to become familiar with the technology which will then be useful throughout the rest of the course. As the group activity was linked to assessment, it also meant students had to get used to coming online, and communicating with others. They would thus be able, and willing, to ask for help and advice through the rest of the course from conference moderators, but more importantly from each other. We wanted to establish a tutor group and course community early on so that students could support and engage with each other. 186 Australian Journal of Educational Technology, 2000, 16(2) Chart 6: Student contributions to national conferences The group work was linked to assessment, but in fact only the student’s analysis of how the group had performed was assessed, so even if the group did not function well it did not affect their performance. What was important was to give them a taste of what group work via this medium was like, and thus to consider the possibilities of the medium. This meant they were forced to become aware of communication issues early on in the course. Due to an interesting discussion about tutor group participation in the national conference, we are changing the size of the tutor group from approximately 15 students to 20. Whilst 15 may be enough to maintain a viable face to face group, a CMC group may need slightly more to retain its ‘critical mass’ of contributors. In conventional UKOU courses attendance at face to face tutorials varies but averages about 50%. Many students, for a variety of reasons, can never attend tutorials. Online tutoring allows all students to participate in the tutorial experience. This was illustrated by the following student’s comment: “I am hard of hearing so this way of studying and keeping in touch with other students is ideal for me as face to face tutorials would cause me obvious difficulties. In fact, the main reason I was attracted to T171 is precisely because the whole course is conducted online, thereby giving me the freedom to participate in group activities by conference messages instead of sat with lots of people trying to figure out who is speaking to whom about what!” Mason and Weller 187 In the survey of students who withdrew from the course, the reasons which were given as the main cause of dropout were increased work or family commitments. This is the same across all UKOU courses. Lack of face-to-face tutorials on the course was not cited. In fact, many students were initially hostile to the idea of group work, but later saw the benefit of it, and requested more group work later in the course. It is also true that many students simply did not like this method of study. They much preferred the traditional UKOU course, with printed course material, face to face tutorials, summer schools and so forth. While online tutoring is particularly appropriate for this course, it may not be the case for all courses. With a modular degree structure students can experience both types of course presentation, and may well find that they prefer one form over another. Interestingly many tutors reported that they had developed closer relationships with the tutor group than on conventional courses because of the greater frequency of communication. As with the training aspect, this will be an area where we can never satisfy everyone. We will continue to use group work as a method of getting students to engage with the technology and each other, but it may be that we need to emphasise even more the nature of the group work, and how it relates to assessment. 5. Tutors – the make or break element of UKOU courses? There is evidence from student surveys conducted by the Institute of Educational Technology going back over nearly thirty years that the support and guidance of the tutor is a crucial component in students’ satisfaction with their learning experience. It is hardly different in face-to- face teaching and studies continue to conclude that technology-based courses are not teacher-independent, despite the hopes of the politicians and accountants (Bates, 1995). It is not surprising, therefore, that a course without face-to-face tutorials, trying to teach IT skills and expecting students to work collaboratively, is going to rely heavily on the quality of its tutors for the satisfaction and success of its students. Of course there is a whole category of adult distance learners who just want to get on with the materials in their own time and who rarely, if ever, contact their tutor. Another category, of about equal numbers, can’t get enough input from their tutor. They constantly request more tutorials, 188 Australian Journal of Educational Technology, 2000, 16(2) more tutor comments in online conferences, and more teaching, more controlling of overly talkative students and faster responses to emails. This course was no exception. Some tutors obviously provided exceptional support: prevented students from dropping out and made the course very enjoyable for others (Charts 7 and 8). Feedback from tutors on this, as on other OU courses which use computer conferencing, shows that tutoring online is perceived as more time consuming and that students are more demanding than on traditionally tutored courses (Mason, 1999). In anticipation of this, the course team prepared a range of materials for tutors (eg. suggested activities for their tutor group conferences, some mid-course review materials and information to use in advising students what follow-on courses were available after completing this course). Tutors found most of these materials very useful, but they did not reduce the overall workload of tutoring the course. Chart 7: Comments on tutor performance It is evident from some of the student feedback that a number of tutors put little effort into moderating their tutor conferences (Chart 9). In some cases, students of such tutors got on with group working despite their tutor’s absence, or they gravitated to other conferences where help and advice was available from central staff or from other students. In the year 2000, more than 12,000 student have registered to take this course – such is its success and its growing reputation. However the demand to find good tutors with the necessary online skills may outstrip Mason and Weller 189 the supply. What lessons have been learned from the first presentation about the kind of support students are seeking from tutors on this course? What steps can you take to help tutors manage the workload? Chart 8: Student view of tutor feedback Chart 9: Use of tutor group for module two discussion 190 Australian Journal of Educational Technology, 2000, 16(2) Course Chair's response Many other OU courses are tutored at least partially online, so there is considerable experience of the medium amongst tutors and academic staff (Morris et al., 1999; Carswell et al., 2000) Many tutors reported that T171 was more time consuming than other courses. This was particularly true during module one, when many students are encountering their initial problems, and as an activity based module it requires greater input from the tutors than the other two modules. We have reduced the content of module one slightly for the 2000 presentation and reduced the amount of work involved in the assignments in recognition of this. In order to support tutor we provide them with a number of ready-made activities, which they run at given times. There are also a number of support conferences for students, for example help with producing web pages, as well as the discussion conferences. Thus the tutor is not the sole source of help on the course. Working via this medium in some ways requires more self-discipline than the traditional face to face tutorials. It is easy for tutors to start answering every question immediately, and soon find they are checking for messages several times everyday. At the other end of the spectrum are tutors who keep putting off checking messages in their conference, which would not be the case for attending a scheduled tutorial in a regional centre. Again I think it is careful to frame expectations of students. The T171 tutors are only appointed part-time and so cannot act as computer technicians for students. Their role is to support the student and the academic material. We are very explicit in stating that it is the student's responsibility to have a functioning computer and Internet connection. The immediacy of the medium can sometimes lead to unrealistic demands on tutors. Some tutors arranged specific days of the week when they would check for messages, so that students knew that they may not get an immediate response, but they would get one by a specific day. I feel that once tutors adapted to the method of working they could accommodate it easily within their normal working routine, and in many ways it was less disruptive than face to face tutorials and all they entail. The UKOU has a very well established staff development and monitoring process for traditional tutorials, and this practice has begun to be transferred to this medium. However, both for those involved in the staff Mason and Weller 191 development process and the tutors themselves, there are new sets of skills to learn and working practices to adopt. For 2000 there is some initial staff development which will occur before the course starts to cover technical issues such as using the conferencing software, and good moderating practice as well as the traditional material about the role of the tutor. There are also experienced tutors appointed on a regional basis to run course and technical support conferences throughout the year, where tutors can gain advice. I feel the experience gained on the pilot study last year will help us provide support to tutors this year, and many of the systems which were bedding down last year (such as the electronic assignment system) are now less problematic. We have a hierarchical support structure in place, and new tutors are being mentored by those from last year. 6. Time – the new distance It has become a commonplace to note how busy people are and how time has therefore become a precious commodity. With the advent of telecommunications technologies, distance is less a barrier to education than it was before the networked personal computer. In fact, it is hardly an exaggeration to say that time is now the barrier that distance used to be in higher education. Chart 10: Time spent studying modules 192 Australian Journal of Educational Technology, 2000, 16(2) Related to this, it is common in OU surveys for students to complain that courses take them longer to study than the 10-12 hours per week expected on a full credit course. It is particularly common for students to complain about the workload on new courses in their first year of presentation. This course was, again, no exception to the rule. See Chart 10 and the following comments: “It is supposed to be a level 1 course, but I am spending three times as much time as other level courses.” “I am enjoying the course but am finding it far more difficult than I anticipated and more work. I have just finished a BscHons with the OU and this is more difficult.” “I am enjoying the course – I just need 36 hour days!” Students new to computing said they spent three, four and five times longer, especially in the early weeks of the course. In later modules, many student complained about the amount of reading, especially reading from a monitor. There is general agreement in the tutor feedback that module one is heavy going, as is the first assignment (Chart 11). One student summed up the views of many: “I am finding this course enjoyable in the way I would find bungie jumping enjoyable. I mean, you must just close your eyes and go, and when you touch down and achieve something, it feels great”. Chart 11: Comparison of workload across modules Mason and Weller 193 Can we put these comments down to the usual first presentation hiccups? Or is the teaching of IT skills, especially at a distance, always going to be frustrating, unpredictable and more time consuming than other traditional subjects? This course combines a number of elements notorious for taking large amounts of time: • browsing the web • interacting in computer conferences and working in groups • getting to grips with a personal computer. What response do you have to students’ complaints about overload on this course? Course Chair's response Module one in particular was somewhat overloaded in that, as an activity based module, many activities (particularly computer related ones) can take much longer than anticipated, particularly if you encounter a problem. As I have mentioned we have taken steps to reduce some of this load in module one for the 2000 presentation. You are correct in that many activities included in the course, such as searching the web, using computer conferences can be quite open-ended activities. New learners in particular can spend a disproportionate amount of time on such tasks. However, the course has clearly been successful in teaching IT skills at a distance, and this is evidenced by the work the students produce at the end of the course, and the feedback they have provided. For the 2000 presentation instructions to students have been refined further and several time-saving options made available for them, for instance the provision of a standard template they can use for creating their group web page. This returns us to our initial point in some ways, that of the need to retain academic credibility. Academic credit is awarded for the nature of the task, not the time taken. So for instance, if a student has spent a long time reading conference messages from other students, whilst this may have helped their understanding of the concepts, it is not activity which replaces the core course work. The amount of time put in by students should not always be viewed as a negative factor. In many cases this reflects their enthusiasm for the course 194 Australian Journal of Educational Technology, 2000, 16(2) and the web in particular. This was often exhibited in the resources they had located for the assignments and the design work put into them. This was not necessary to complete the assignment, but in doing so the student made the learning experience more rewarding and meaningful. UKOU students often have busy lives apart from their study, and fitting in a course can be difficult. This is particularly true of entry-level courses, where many students are new, or returning to study. The course contained a lot of material on developing study skills, such as critical reading, clear thinking, effective writing and so on, but this is maybe one area where we could help students further by giving them advice on the sorts of 'survival tactics' students tend to develop after one or two successful courses. Workload is a factor we will have to continue to monitor; we also need to balance carefully the feedback from students and the academic demands necessary for a level one undergraduate course. 7. Using student feedback in revising the course Reading though all the feedback data from students and tutors is like standing at the apocryphal Spaghetti Junction and watching cars going every which way. Some students call for more group work; others want none at all. Some are disappointed in the course content; others find it the perfect marriage of both vocational and academic skills. Advice fumes the air. At a general level, several common responses do emerge however: • that students have enjoyed the course tremendously • that it is a lot of work for both tutors and students • that the systems, procedures and framework of the course have functioned successfully (eg. the web site, the electronic submission of assignments, the computer conferencing, and the course content generally) So do you tweak the most unsatisfactory elements (eg. reduce workload, improve materials for new computer users) and sit back and bear the inevitable criticism from the students who do not like this or that element? If so, what value have you and the course team had from the evaluation data? Mason and Weller 195 The course is innovatory for the UKOU in a number of ways: • there is no provision for face-to-face tutorials • the course content, apart from several set books, is entirely on the web • the combination of ICT skills teaching to complete beginners, online group work, and very large scale online delivery (even by OU standards) is probably unique in the world. How successful do you consider the course to be in achieving its original aims? Course Chair's response The most consistent feedback was that the preparatory period needed to be longer and that module one was overloaded, both of which have been addressed for the presentation in 2000, as well as the inclusion of a non- assessed group activity in module two. As you indicate, apart from these it was often difficult to discern a clear message. From the feedback of the students who completed the course, there is no doubt that the great majority found the course enjoyable and rewarding, for example 84% of respondents rated module two as enjoyable or very enjoyable. What this indicates is that we need to be very clear to students what the course is, and as importantly, what it is not, about. The course descriptions have been modified now, to reflect some of the feedback from last year. Also the course guide, which all students receive in the mailing, states very explicitly a number of factors students need to be aware of pertaining to the course, including group work, the academic nature of the material, the teaching of study skills and the need to be online for substantial amounts of time. At this stage students can still withdraw from the course. The feedback has been particularly useful in helping us modify specific activities. Perhaps more importantly it helps the course team form a picture of the students on the course, the issues which are important to them and the manner in which such a course can impact upon their lives. 196 Australian Journal of Educational Technology, 2000, 16(2) In some of its aims the course has exceeded reasonable expectations, for instance in appealing to so many people, often new to studying and new to the OU. Some aspects we have only begun to explore, and will develop over the years. For instance, we can assume 100% attendance in asynchronous tutorials, and discussion is easy to generate in this medium, so the underlying pedagogy could shift towards a more discursive approach. One of the most interesting aspects this year was the range of quality of the assessments students produced, particularly their end of course assessment. This is one area where I believe the possibilities offered by the medium will make a significant difference to our standard practice. For instance it makes plagiarism easier, and at the same time it encourages students to find their own resources, to think about the design and structure of their answer, and perhaps to engage with the material in a different manner. The UKOU has developed a model of distance education which works very well, and it has a reputation for good quality materials. Perhaps the main aim of the course team was to adapt this effectively to the Internet. The student and tutor responses this year, and the huge demand for the course in 2000 indicate that we have largely been successful in this aim. Given the nature of the material, and the ease with which it can be updated compared to traditional print material, the course will continue to change throughout its lifetime. Web courses at other institutions How typical are the benefits and the difficulties we encountered on T171 compared with the experience of other institutions? In order to investigate this question, the results of three evaluations of web courses at other institutions are compared. The first example is given by Hara and Kling (1999) of Indiana University in a paper significantly entitled, ‘Students’ Frustrations with a Web-based Distance Education Course: A Taboo Topic in the Discourse.’ The authors point out that ‘many advocates of computer-mediated distance education emphasise its positive aspects and understate the kind of work that it requires for students and faculty.’ In their evaluation of a small web course offered by one of the major US universities, they uncovered a range of frustrations and difficulties students faced which actually inhibited their learning at both conceptual and affective levels. The main problems they identify were: Mason and Weller 197 • lack of prompt feedback by the tutor • ambiguous instructions on the web • technical problems. Furthermore, they noted that students’ expectations about the course affected their satisfaction with it, when the materials or instructions diverged from their pre-conceptions. In a second example, a graduate health services planning and policy course at California State University, Bakersfield, the evaluator (Alexander, 1999) reveals that promoting collaboration amongst the students was more problematic than anticipated and concluded that. IT used in an exploratory/constructivist model provides excellent opportunities for collaboration . . .provided care is taken in promoting collaboration, and in presenting and structuring assignments around these communities. (Alexander, 1999, p. 22) Other important findings were that: • students need practical and applicable IT instruction and guidance in the learning process • acquiring new IT skills is a time consuming task for students. (ibid) A third evaluation of a web course comes from the University of British Columbia and concerns a graduate course on distributed learning. The evaluators, Bartolic-Zlomislic and Bates (1999), found that their course was overloaded and the work had to be reduced in subsequent presentations. Adjustments also had to be made to the conference structure and small group sizes, and the collaborative assignment provoked mixed reactions from students. Some students would have liked a more applied focus to the course and others complained of poor moderation, unanswered questions, and intimidation. Nevertheless, the course was very successful overall and the value of online, asynchronous interaction, access to web resources, collaborative work and ease and flexibility of the web as a delivery medium created a rich learning environment appreciated by most of the students. 198 Australian Journal of Educational Technology, 2000, 16(2) From this short comparison with other web courses, it is clear that the problems and the advantages encountered by the UKOU in its web course are in no way unique and seem to be independent of the numbers of students, which range across the three external courses and the OU course from six to 900! Conclusions We have examined this innovative course from the students’ perception of the issues raised by web-based teaching. The feedback from students indicates that the main issues were: • the time it took to become competent with the PC, the Web and/or with computer conferencing • the sense of accomplishment and satisfaction with the course and the experience it provides of the whole ICT world • the appropriateness or not, of teaching ICT skills and of working in online collaborative groups. The factors which most affect students’ satisfaction relate to: • the support of their tutor or other staff or students • the amount of time, patience and motivation they have to devote to the course • the extent to which the course content and presentation fit the students’ expectations and learning style. These findings are not inconsistent with the findings of other evaluations of web courses at other institutions. Bibliography Alexander, J. (1999). Collaborative design, constructivist learning, information technology immersion, and electronic communities: A case study. Interpersonal Computing and Technology, 7(1-2). http://www.emoderators.com/ipct- j/1999/n1-2/alexander.html [verified 14 Jul 2000] Bartolic-Zlomislic, S. and Bates, A. W. (1999). Assessing the costs and benefits of telelearning: A case study from the University of British Columbia. http://research.cstudies.ubc.ca/nce/ubcstudy/title.html [verified 14 Jul 2000] Mason and Weller 199 Bates, A. W. (1995). Technology, Open Learning and Distance Education. Routledge, London. Carswell, L , Thomas, P., Petre, M., Price, B. and Richards, M. (2000). Distance education via the Internet: The student experience. British Journal of Educational Technology, 31(1). Cringely, R. X. (1996) Accidental Empires. Penguin, London. Hafner, K. & Lyon, M. (1998) Where Wizards Stay Up Late: The Origins of the Internet. Simon & Schuster, New York. Hara, N. and Kling, R. (1999). Students’ frustrations with a web-based distance education course: A taboo topic in the discourses. http://www.slis.indiana.edu/CSI/wp99_01.html [verified 14 Jul 2000] Hodgson, V. and McConnell, D. (1995). Co-operative Learning and Development Networks. Journal of Computer Assisted Learning, 11(4), 210-224. Laurillard, D. (1998). Multimedia and the learner's experience of narrative. Computers and Education, 31, 229-242. McConnell, D. (1994). Implementing Computer Supported Cooperative Learning. Kogan Page, London. Mason, R. (1995). Using electronic networking for assessment. In F. Lockwood (Ed), Open and Distance Learning Today. Routledge, London. Mason, R. (1998). Models of online courses. Asynchronous Learning Networks Magazine, 2(2, October). [verified 14 Jul 2000] http://www.aln.org/alnweb/magazine/vol2_issue2/Masonfinal.htm Mason, R. (1999). IET’s Masters in Open and Distance Education: What have we learned? http://iet.open.ac.uk/pp/r.d.mason/MAEval.PDF [verified 14 Jul 2000] Morris, R., Mitchell, N. and Bell, M. (1999). Student use of computer mediated communication in an Open University Level 1 course: Academic or social? Journal of Interactive Media in Education, 99(2). http://www- jime.open.ac.uk/99/2/ [verified 14 Jul 2000] Salmon, G. (1999). Computer mediated conferencing in large scale management education. Open Learning, June, 45-54. Thorpe, M. (1998). Assessment and ‘third generation’ distance education. Distance Education, 19 (2), 265-289. Tiberius, R. (1999). Small Group Teaching. Kogan Page, London 200 Australian Journal of Educational Technology, 2000, 16(2) Wegerif, R (1998) The social dimension of asynchronous learning networks. Journal of Asynchronous Learning Networks, 2(1). [verified 14 Jul 2000] http://www.aln.org/alnweb/journal/jaln_vol2issue1.htm#Wegerif Weller, M. (1999). A large scale Internet based course for computer beginners. In Collis, B. & Oliver R. (eds), Proceedings EdMedia 99, Seattle, Washington, USA. AACE, Vancouver. Weller M. J. and Hopgood A .A. (1997). Implementing a learning model for a practical subject in distance education. The European Journal of Engineering Education, 22(4), 377-389. Robin Mason is Head of the Centre for Information Technology in Education at the Open University. She is also Director of the Masters Programme in Open and Distance Education. Dr Robin Mason, IET, The Open University, Walton Hall, Milton Keynes, MK7 6AA Tel: 01908 653137 Fax: 01908 654173 Martin Weller is a lecturer in Telematics at the Open University and chair of the course T171: You, your computer and the Net. Dr Martin Weller, The Open University, Walton Hall, Milton Keynes, MK7 6AA. Tel: 01908 652915 Fax: 01908 653658