I NV ESTIGATING TH E EVALUATION P ROCEDURES FOR A DISTANCE LEARNING UNDERGRADUATE DEGREE IN LIS by Susan D. Lithgow, Clare Thomas and Marianne Taylor Abstract This article is based on a process evaluation carried out at the Department of I nformation and Library Studies (DILS) at the University of Wales Aberystwyth (UWA). The research investigated the effectiveness of the existing evaluation procedures for the BSc Econ in Information and Library Studies by distance learning. After briefly discussing the research process, the article then describes the findings in relation to the perceived gaps in the evaluation procedures, including the importance of disseminating the evaluation results. I t was found that changes to the course occurred in response to feedback from various sources and that evaluation data was used in ways that differ from their perceived purposes. Finally, the article relates these findings to the different concepts of quality found to be appropriate to a BSc ILS Distance Learning programme taught both on and off-campus. The Evaluation Context The Department of Information and Library Studies (DILS) at the University of Wales Aberystwyth (UWA) has substantial experience in providing distance learning courses. I nitially this was at post-graduate level, but since 1 993 it has also run an undergraduate programme (BSc Econ ILS) by distance learning, for students working within the profession who do not have a qualification at the professional level. For these students the usual full-time paths to gaining such a qualification present geographical, financial and other barriers. The student profile is predominately female with students working in a wide range of information organisations such as public, academic and school libraries as well as business and media organisations. The Open Lear n i n g U n i t (OLU) at DIL S UWA was established in conjunction with the BSc Econ ILS programme, but is now responsible for the administration of all the DILS distance learning programmes and for supporting staff and students involved in them. Staff are employed in the development of programmes both by working with academic staff within DILS to produce materials in a format suitable for distance learning, and by commissioning outside experts in the field to produce specialist modules. The unit also develops other media to support learning both on and off campus, such as a computer conferencing system. Finally staff are responsible for training throughout the University and beyond and actively involved in research within the field of distance learning. Currently the OLU has ten full and part-time staff members and supports 4 1 4 distance learners, 249 of whom are BSc Econ students. The Evaluation Rationale and Objectives The rationale for the evaluation discussed in this article arose from several factors, both i nternal and external, including the innovative nature of the B Sc Econ ILS by distance learning and the desire to expand the research base of the OLU in conjunction with DILS. Externally there have also been recent developments that justified research in this area. The Quality Assurance Agency (QAA) is responsible for the assessment of teaching quality in the higher education sector in the UK. It has recently proposed a shift from a series of single event inspections (quality assessment) to a focus on universities' own review processes (quality audit) (QAA, 1998). The Dearing Report in the UK on Lifelong Learning has drawn attention to issues concerning widening participation to higher education for adults including those in the workplace E DUCATION LIBRARIES VOL. 24, Nos. 2/3 s A R T I C L E S (Dearing, 1 997). Finally, there is a global interest in developing distance learning programmes, and within the LIS community it has also been posited as an alternative means of delivering education to information professionals at all levels (Fasick, 1 995). The remit of this research was to try and identify those aspects of evaluation which were of concern to the users of the existing evaluation procedures in order to create a more cohesive system, relevant to the needs of those users and future users. The research team began by identifying the users of the current evaluation procedures and proceeded to investigate: o users' perceptions of the existing evaluation procedures o the effectiveness of the existing evaluation procedures o the identification of gaps in coverage in existing evaluation procedures . Methodology As a process evaluation (Patton, 1 980) the research demanded a mainly qualitative approach since the evaluation process, as opposed to the outcomes of the course, was being examined. Moreover, it was hoped that findings from the research project would allow the unit to understand some of the perceptions of the users of the evaluation procedures and the effectiveness of the present quality assurance system. It would also provide qualitative information about any gaps identified and contribute to the SQecific content of any future evaluation tools . In this respect, it was also essentially a self-evaluation as defined by Calder ( 1995) and Harris and Bell ( 1 986) in which a department or unit reflects on its present practices with a view to (if necessary) developing them in a way that is more appropriate to meeting the expressed needs of both its internal and external users. Although the research noted the difficulties about validity when using mixed approaches to evaluation at a theoretical level (Jakupec and Kirkpatrick, 1 998) nevertheless the l iterature revealed that it was legitimate in the context of an in ternal evaluation to use both quantitative and qualitative methods (Morgan 1 984; Dillon and Gunawardena, 1 992). The fact that much distance education research and evaluation is bounded by context was seen as pertinent to research in an i nnovative area of ILS education and in the particular circumstances of a programme taught at a distance from within a traditional university department. The research therefore relied upon a series of complementary methods of data collection to find out the views of students and staff including: o Literature Search: This was carried out in various areas, for instance, quality in h igher and distance education, the concept of dual-mode universities, and student feedback. o Focus Groups a n d I n terviews : These provided a n illuminative dimension t o the research and the results also influenced the production of the questionnaires. EDUCATION LIBRARIES VOL. 24, Nos. 2/3 o Student and staff questionnaires: These covered the respondents' perception of evaluation and their experiences of the evaluation procedures including the identification of perceived gaps. o 'Brown papering' of procedures and observation of evaluation sessions. In relation to identifying gaps in the evaluation procedures, the research was interested in ascertaining how quality was perceived in distance h igher education and i n locating any performance indicators that could be applicable to a programme which is taught to distance learners from within a conventional campus-based department. Rumble terms universities which teach in this way as 'dual mode' and in many ways DILS UWA operates as a dual mode department (Rumble, 1 992). The research team was aware of the debates about the difficulty of defining quality in higher and distance education. As many commentators have pointed out, quality in education is a contested concept (Green, 1 994). This is related to the difficulty of transferr i n g from i ndustr ial and managerial domains, termi nology such as quality assessment, quality control, and quality assurance which are thought inappropriate to measure the complexity of a process such as higher education (Barnett, 1992; Tait, 1 993). In terms of the performance indicators for distance education the literature search revealed some key publications, such as the European based project, the SATURN Quality Guide for Open and Distance Learning (SATURN, 1 992) and within the UK, the Manpower Services Commission's handbook, Ensuring Quality in Open Learning (Manpower Services Commission, 1 987). However whilst these were useful, they were found to be very wide in scope and concerned with providers of distance education in a variety of situations such as work-place training as well as education at different levels . Other works by Calder ( 1 994a), Thorpe ( 1 993), and Rowntree ( 1 992) were useful in identifying the key areas of evaluation for distance education in the university sector. The recent UK Quality Assurance Agency Guidelines on the Quality Assurance of Distance Education were published towards the end of this research . For the BSc Econ at UWA, the academic and professional evaluation of the programme lies with the quality procedures for all courses validated by the university and also by the professional accreditatio n bodies concerned, which are the UK Library Association and Institute of Information Scientists. The official UWA quality document contains references to support for distance learners and a distance learning quality checklist is distributed to Heads of Department, to remind departments involved of the quality indicators required at UWA. From within these broad contextual issues of qualily in d is tance h igher educati o n the research team devised the focus-group schedules and i n the light of those findings, the student and staff questionnaires. Five focus groups were held at the annual A R T I C L E S 1 residential school, representing a sample of all the BSc student intakes present. The subsequent student questionnaires were sent out to . all 222 BSc Econ distance learning students and 82 were completed and returned-a response rate of 37% . The staff questionnaire was sent out to all 30 staff in DILS and 20 were completed and returned-a response rate of 67%. The subsequent results from these, together with the results from the other methods described above, led to the emergence of a number of themes. Given the mainly qualitative nature of this research, the team depended on the focus groups and the open questions to give an indication of the intensity and priority of particular issues of concern to users. The remainder of this article will focus on the perceived gaps in the existing evaluation procedures as demonstrated by the research and consider how they might relate to issues of quality and evaluation in distance education. It will then go on to look at the relationship between the perceived purposes of the evaluation procedures and their actual use. Finally some of the issues this raises for staff and students concerned will be addressed. Findings: Perceived Gaps in the Existing P rocedures Before looking in more depth at the gaps identified by the research, i t should be noted that key evaluation procedures were found to be in place. For example, the evaluation of the module materials and of the residential schools was well established. These occur firstly through the Module Evaluation Form (MEF) which is placed at the end of each module and which students can return with their assignments to the Open Learning Unit. The evaluation of residential schools takes place at the end of the week in a face to face evaluation session between staff and students. It starts with a set agenda, covering study school administration, on-campus IT and Library Services, counselling and teaching, but the format also encourages students to raise any aspect of the course that they wish. It should be noted that the majority of users considered that these, together with other informal opportunities, provided adequate means of evaluation-espec i al ly from a student viewpoint. In spite of the above, about a third of users, both staff and students, identified gaps in the existing evaluation procedures in response to direct survey questions or as a result of focus-group discussions and interviews. The gaps identified in the evaluation procedures were broadly categorised as: 1 . Organisation-Related - those concerned wi th the evaluation of student support at a distance. 2. Course-Related - those concerned with the gaps in the evaluation of the wider teaching process. 3 . Student- Related - those concerned with the student learning experience and environment. 4. Process-related - those concerned with the dissemination of the evaluation data. Inevitably these categories are not mutually exclusive and indeed the findings supported the view that in the context of distance education, educational, administrative and supportive processes are necessarily interwoven. In looking at the perceived gaps in this way, it is evident that they are principally concerned with the needs of students when studying away from the campus and with students and staff involved in educational and administrative processes which are characteristic of distance education. Moreover, the research showed that the gaps identified were often related to processes that are ' intangible' and difficult to measure. In this respect they reflect the concerns with quality issues that were identified in the l i terature search as particularly important to distance education. The issues related to each of the four main categories will now be discussed in turn. (1) Organisation-Related Issues These were seen as quali ty issues that related to the support provided by the organisation for students when studying at a distance and comprised the evaluation of: • OLU support and information provision • the personal tutorial system • access to academic tutors • resources such as Library Services. It has been shown that support for distance learners will be interpreted in different ways by different institutions. This can inc l ude both academic a n d perso na l counsel l i n g , and administrative services or may be focused on particular aspects of the overall provision. (Robinson, 1 995). In the case of DILS UWA, the Open Learning Unit undertakes all of the administrative work for delivering materials, dealing with finance, responding to enquiries, and circulating information . S tudents are also assigned a personal tutor from Open Learning Unit staff with whom they meet at every study school and are encouraged to contact whenever they are experiencing difficulties or for general guidance when away from the campus. The unit informs students of their progress through the course at six-monthly intervals and there is a lunchtime help-line every week. While many students identified the various ways of contacting the Open Learning Unit (e-mail, computer conferencing, telephone, letter) as opportunities for informal ad-hoc evaluation of the course itself, they did not identify any formal means of evaluating these services unless they arose for some reason at the evaluation at the end of the residential school. Students are also encouraged to contact their academic tutors either directly or through the OLU but again this service is not formally evaluated. The identification of these particular gaps can be related to issues of support for students from the organisation when they are away from the campus. E DUCATION LIBRARIES VOL. 24, Nos. 2/3 a A R T I C L E S The evaluation of supporting distance learners in these ways is obviously not easily measured. Apart from efficiency indicators concerning speed of response, they can also be related to more intangible human processes, which are often complicated by the distance involved. Students and staff are both employed in other duties. While a phone call will often provide the most satisfactory and speedy response, i t was suggested by some focus group discussions that it can also be difficult to judge the convenient moment and that this will effect the quality of the interaction. Students also gave examples of kinds of difficulties inherent in measuring consistency of response, which will naturally occur when staff attempts to provide adequate support at a distance. Two contrasting student comments were: I have phoned on occasions when I was seeking clarification of something and sometimes it's useful and sometimes it isn't, depends on who you get on the other end of the phone to be honest. I was unsure about the Information and Society module, so I e-mailed, but he rang me up and spoke to me and sorted me out. The identification of these gaps in the evaluation procedures suggests that users were concerned with quality not only in terms of the kind of information they receive but also in terms of timeliness and consistency of response. This would equate with the ways in which some distance learning experts link the provision of support for distance learners to the quality criteria used by service management theory (Sewart, 1 993 ; Ljosa, 1 993) . Ljosa for i nstance, l inks the intangible interactions of service delivery with the processes of education and with the focus in distance education on students and delivery systems. Such interactions are compared to what such theories refer to as 'the moment of truth' when a customer experiences the value of what is delivered. These moments are often intangible and ephemeral but are also crucially part of how students value the institution. The provision of library services is another area that has been increasingly recognjsed as important in the support of distance learners (Unwin, 1 998). In fact the provision of a postal service from the campus library to distance learners is another innovative dimension to this course. Just before the research began, the library had carried out a user survey of its services to distance learners, and the provision of on-campus faci lities is evaluated at the end of each study school. However, some users did identify the lack of an on-going formal evaluation of these services as a gap in the evaluation procedures. EDUCATION LI BRARIES VOL. 24, Nos. 2/3 The importance of finding ways of providing and evaluating the support provided for students at a distance has been one that Sewart ( 1 993) has demonstrated as being critical to the success of educational programmes. Moreover, these gaps would seem to accord with Sewart's contention that student support acts as an adjunct to the teachjng materials in allowing students to experience the discrete elements of the process as a whole. (2) Course-Related Issues These were quality issues which lay outside the evaluatjon of teach ing materials but which were part of the wider teaching-learning process. These comprised: • assignment turnaround times • assignment feedback in terms of consistency and quality of feedback comments • clarity of instruction around assessed tasks • modules in terms of their relevance and up-to-dateness. The BSc Econ distance learning programme provides a large amount of flexibility with regard to setting assignment deadlines. Students can decide their own pace within the minimum target of completed modules per annum. In distance education, the provision of flexibility with regard to course structure can provide more choice to students and may in itself be an indication of the quality of a course in terms of 'openness' . However i n a dual-mode situation, this can add to administrative complexity and put pressure on staffs who are working to two different schedules. In order to create a more coherent pattern for both students and staff , the OLU and OILS have set a number of marking dates that span the academic year. Student assignments are held until these dates and handed over to staff as assignment batches. The nominal assignment return period is one month, including administrative procedures. The OLU monitors this process and provides on-going information to students in cases where there is l ikely to be any delay in the return of marked assignments. The research revealed that students wished to have the opportunity to evaluate assignment turnaround time. Staff also felt that assignment turnaround was important and identjfied it as a quality issue particularly in terms of focusing on the student as stakeholder but did not cite it directly as a gap. In contrast to other perceived gaps in the evaluation procedures, student respondents often commented more extensively on the reasons for their dependence on timely feedback from assignments and this gave the researchers an insight into why this was an important quality issue for student respondents. They did so in terms familiar to the evaluators of distance education. In distance education, A R T I C L E S s assignment feedback is part of the important process of creating a ' learning dialogue ' . The speed and quality of feedback also contributes to student motivation and reassurance, particularly at the start of the course when there is a concern about 'being on the right track' and towards the end when students are anticipating their final result. The speed of assignment turnaround is often cited as a crucial performance indicator for higher education in general and because of its nature for distance education in particular. However, it has also been asserted that the speedy return of assignments does not i n i tself guarantee the quality or consistency of the feedback in terms of teaching. Again this has been shown to be particularly important i n correspondence teaching (Thorpe, 1 993). As Helen Len tell ( 1 995) states, 'To return student assignments quickly is important, but is it the same as giving students the kind of feedback they need in order f or them to learn and progress' (p. 1 22). Once again this moves the focus of evaluation from the easily measured to the more complex interactions of the process of learning and the results of the research confirmed that students were also interested in evaluating the quality and consistency of assignment feedback. In relation to this, a comment made in one of the focus groups could also be connected to the issue of timeliness of evaluation and how some students perceive the purpose of evaluating the learning materials as part of the process of learning which is not complete until assessment feedback has been received. The student seems to move evaluation from the product or learning materials to the process of the teaching-learning transaction. It may also be related to the evaluation of the clarity around assessment tasks. This is in itself an area in which the OLU provides on-going support that extends the work in the modules to an on-going activity-again, a process rather than product. As a student commented: But you don't always realise that you misunderstood it until it's too late-we evaluate the module after we've just done it and it says "was the question clear "and you say "yes" because you think you've understood it-then you send it off, get the feedback and you're told that you've misunderstood it so then you could do with evaluating it at that point, and also you'd be able to evaluate the marking because one assignment came back with very useful comments and I appreciated them but there was no point to say this. The measurement of assignment turnaround in a dual mode context is a question of balancing the different aims of the course in relation to providing f lexibil ity f or distance learners and enabling staff to provide a coherent response across a cohort when marking. For a significant minority of student respondents, the flexibility of the course in relation to submission of assignments and the ability to work with a large measure of autonomy were more !mportant indicators than that of assignment turnaround. As one student respondent said: To be honest I do not pay particular attention to the marking or return dates. I feel that this is just an added pressure, and would rather see my work completed to my satisfaction, than rush to get it in by a marking date. As already discussed, at the end of each module the OLU has provided a module evaluation form (MEF) and this covers many aspects of the learning process particularly in relation to the module materials. This was identified as the main format for the summative evaluation of the module that a student has just completed and which can be sent in with the assignment. The majority of users felt that the MEF was a satisfactory format for evaluating the course materials. However, in spite of its comprehensive nature, a number of staff and student respondents agreed that the MEF did not cover some aspects of the course materials that they thought important to evaluate, such as the relevance and currency of materials used in the modules and module revision schedules. The problems of relevance and up-to-dateness are referred to by Cresswell and Hobson , ( 1 996) as one of the areas in which (particularly for distance education) misunderstandings of the purpose of the materials included in a course may occur between students and authors. They point out that this often occurs in professional education. The fact that this was perceived as a gap in the evaluation of the B Sc by a number of students and staff would seem to confirm this view. However, while Cresswell and Hobson show that authors and students can have opposing points of view in relation to the relevance and up-to-dateness of teaching materials, this research found that it was an issue that both staff and students would regard as important to evaluate. Perhaps this is due to the nature of the profession, which is one where academic staff and those working in the f ield cannot ignore the rapid changes taking place. EDUCATION LIBRARIES VOL. 24, Nos. 2/3 1 0 A R T I C L E S (3) Student-Related Issues These gaps relate to quality issues that were concerned with the environment in which dis tance learners study. This is sometimes referred to as the 'student learning milieu' and may be of particular interest to providers of a professional LIS course undertaken by those simultaneously working in the field. These were: • employer support • student experience in a learning environment • longitudinal evaluation. For some students and staff the ways in which the course interacts with the workplace was seen as an area that might be subject to evaluation . For instance, a small number of users identified the attitude of employers and the different levels of support that students receive in the workplace as a gap in the evaluation procedures. It also arose in focus group discussions and was clearly of i nterest to students, who related it specifically to financial support, colleagues' support and their different status within different institutions. In the focus groups there was much discussion around the kind of environment in which the students studied and worked. This was in terms both of the various levels of support that students experienced and on their status as learners currently working at the level of both theory and practice. Students also referred to their working life when reflecting on how they measured their progress, citing the ways in which they could apply the course directly, or how their learning enabled them to view their organisations from a new theoretical perspective. The evaluation of these aspects of the learning process must fulfill the call by some distance learning experts that evaluation and research in distance education should extend beyond the materials, or support systems to the experience of the students themselves (Evans, 1 994 ). In terms of LIS education by distance learning, the impact of studying for a degree and qualification by this method may be of interest in terms of the kinds of skills students are given in the programme and their implementation not only in the future when they move into the professional side of work, but the impact on their present employment and employers. In relation to this, students assumed very often that student destinations formed part of the evaluation procedures. Some staff cited longitudinal data as a perceived gap in the evaluation. The focus group discussions revealed this was not as simple a measure as i t might be for full-time students moving into a new career. For adult learners the reasons for taking a degree may be more varied and cannot EDUCATION LIBRARIES VOL. 24, Nos. 2/3 I always be measured by s imple destination routes. This was evidenced here i n the wide range of different responses that students gave i n terms of motivation for undertaking the programme and ambitions for the future. These ranged from intrinsic motivations such as ' I 'm doing it for myself' to a fourth year student whose extrinsic motivation was such that he was able to project his firm plans to progress from his present position to that of 'County Librarian' . The results also showed that many students would agree that they gained more confidence in their profess ional and personal lives and that what they valued in studying for the B Sc Econ was also an expansion of their opportunities . As one put it: I just think at the end of it, I've just got a choice, that's how !feel about the degree, it gives me a choice and I don't know what that'll be until! get to that time. It's a nice thought. Moreover, the data from the focus groups also revealed that some students were already in professional posts while others had obtained higher level posts during their studies. For instance, one small group consisted of six students who were returning for their dissertation study school, which students can choose to attend either during their third study school or separately towards the end of their studies. It emerged that four of these students had moved into h igher level posts while doing the course and the pressure of coping with demanding new positions had resulted in them taking longer than anticipated to complete the degree. Again this would indicate that in terms of performance measurement and evaluation, the criteria used by students and staff in discussing the value of this cour se is more complex than the straightforward recording of student destinations. It involves a range of relevant variables including present employment, and employer and student expectation. (4) Process-Related Issues· Arguably the most significant gap that emerged from this evaluation was in regard to the process itself, particularly in terms of 'closing the feedback loop' . These were identified as: • access to evaluation data • feedback on evaluation data • dissemination of the results of evaluation. The identification of these gaps emerged in the comments that respondents made in the student and staff surveys and in relation to the findings about the effectiveness of evaluation procedures in terms of the link between evaluation and follow up action. A R T I C L E S 1 1 D i stance educat ion has been described a s ' two-way communication ' , and i t would seem that for these students, eva�uation was also an important means of maintaining a dialogue with the course providers and enhancing their sense of belonging to the institution. For instance, the comments of student and staff respondents demonstrated that they valued the opportunity for contact and immediate interchange of views which the face to face evaluation sessions at the end of study school allowed them. However, when students are evaluating modules at a distance, they are do ing so i n i so la t ion and many identi fied the dissemination of the overall results as a gap in the feedback process. Here too, course evaluation was seen to have a role in creating a sense of belonging to the institution and this was evident in the kinds of reasons that students gave for returning completed module evaluation forms (MEF) apart from those of providing feedback. Evaluation was seen as a chance to express their thoughts and feelings about the course and was evidenced by the occurrences in the questionnaire data of the words 'contact' or 'communication' . It may explain why 80% of student respondents stated they always returned the module evaluation form but only 25% believed that i t led to changes in the modules. A significant number of staff also identified a gap in the circulation of feedback data. They were aware of the difficulties of analysing and disseminating evaluation data that was seen as time-consuming and logistically complicated . In traditional courses, it is often up to staff to devise course evaluations. Evaluation results are seen as a means of private reflection on how successfully the curriculum is delivered. Distance education is much more reliant on formats that can be used across the board and can therefore make useful comparisons. Also the evaluation centres on the extent to which the learning materials facilitate student learning and may be less concerned with content. It was striking that student comments in the survey with regard to the evaluation of modules through the MEFs were directed more to the Open Learning Unit than to academic staff. Staff wished to h ave a more comprehensive, h o list ic evaluation of the course, perhaps by an overall quantitative evaluation. This related to questions of ownership, access to data, and the realistic use of time and resources. In terms of module evaluat ion , staff often required more detai led, qual i tative information and more effective means of making the results available. Action as a result of evaluation feedback is recognised as an important but sometimes neglected phase of the evaluation process. Harvey ( 1 999) states: ' Despite the "customer rhetoric", students are not repeat purchasers of products but participants in a learning process designed to improve their life chances. Feedback will only play a significant role i n empowering students if it leads to action' . In relation to the desire to know what action was taken i n relation to the evaluation o f this course, many students expressed a desire to have ' feedback on feedback' , including information on how evaluation data was used by course providers. About a quarter were able to cite instances of change occurring as a result of evaluation or sometimes in ways that seemingly contradicted evaluation. In particular, the feedback given at the end of study school sessions and concerned with those issues having to do with residential schools was seen as formative and led directly to changes which benefi ted those participating in future years. Examples of th is were given in the survey and i ncluded improvements to accommodation and to pre-course provision, timetable changes, the pace of study school and improvements to library access, the marking system, and IT provision. With regard to changes occurring as a result of feedback through the MEFs, and in common with much student course evaluation, students are not able to directly identify any changes made, since they do not re-take the module. Some students were ·able to cite examples of changes to modules as a result of informal communication at study school. The majority of students felt that feedback through the MEFs should lead to improvements in the modules and assumed that this was the primary role of the module evaluation form. From a staff perspective, it was evident that although the module evaluation data was used, it was more likely to be in ways that went beyond its stated or perceived purposes. For instance, i t was found that the module evaluation forms were used when staff wished to check the reception of the course in a general way. They were used formatively by staff developing new courses. They were also used by the OLU in its role as the i nterface between academic concerns and support issues for students in providing study guides, clarifying assessment tasks, or organising the sessions at study school. However, on the evidence of this research, the use of MEF data to make on-going changes to current modules was found to be somewhat limited. It also seemed that academic staff often used other means of evaluation when judging the effectiveness of current modules. For i nstance, a small number of academic staff respondents indicated that they made changes to modules as a result of their assessment of how students had coped with the modules after they had marked a batch of essays. This is indicative of the kind EDUCATION LIBRARIES VOL. 24, Nos. 2/3 1 2 A R T I C L E S of tacit evaluation that is used in on-campus situations where tutors can modify any changes made as a result of evaluation gained through other contacts with students and may be an example of professional reflective practice. In relation to this, some academic staff also reported that changes were more often made to modules as a result of informal feedback from students by e-mail or telephone. It would seem that they were using these means almost in replication of the kind of instantaneous feedback that they would receive from on-campus students. This might indicate that more change would occur in relation to the MEFs if they were analysed and disseminated rather than being made available for individual consultation as at present. It also suggests that staff is responding in traditional ways to ad­ hoc informal feedback, or, as indicated elsewhere in the research, that in order to facilitate change to modules as result of feedback through the module evaluation form, staff would prefer more qualitative information. These results confirmed the views of some commentators who maintain that change as a result of feedback in organisations is not a simple causal relationship between input and output (Morgan, 1 993) . Moreover, i n their comments on the evaluation of the course in terms of action, staff often cited the pressures of time and resources as limitations on the kind of practice they saw as ideal. The relationship between feedback and subsequent improvements to courses is very much an issue where political, resource, and management issues will exert as much influence as feedback from students. Conclusions: Evaluation of an LIS Course in A Dual-Mode Department The evaluation of programmes delivered in two modes has been linked with the debate about how far distance education remains a distinct mode of delivery and how far it can be said to be converging with traditional education with in universities. Lippiatt ( 1 997) feels that the convergence of both modes can be reflected in the convergence of evaluation criteria in a way that is mostly unproblematic and formulates a range of performance indicators that are relevant to both modes. Others feel that distance education is a more complex process with more to evaluate and more complex criteria (Thorpe, 1 993) . The findings of this research in relation to gaps in the existing evaluation procedures would confirm this latter view. While the central evaluation of the learning materials and on-campus teaching were found to be adequately evaluated, it emerged that there existed a perception of gaps in the evaluation procedures. These dealt with the EDUCATION LIBRARIES VOL. 24, Nos. 2/3 evaluation of the student experience when studying at a distance, their on-going support, learning environment, and the teaching received through assignment feedback. Users were also interested in evaluating the intangible aspects such as the kind of service they received in terms of administrative and academic support. These have less to do with the learning materials and more with the process of learning accompanied by support, administrat ion , and the wider teach ing- learning transactio n . Robinson ( 1 994) has referred to these as the 'conditions for learning' in which 'quality lies in the totality of products, delivery, services and general ethos' ( p. 1 86). Moreover these can be said to be of greater importance to distance education and therefore to be of particular significance in a dual-mode situation. In distance education generally, evaluation itself has become . more process-oriented, both in terms of the administrative and support services that are necessary for a successful distance education programme and a related 'customer-oriented' approach. Morgan ( 1 997) asserts that the impetus for easily measured outcomes and the production of performance indicators for higher education now comes mainly from governmental agencies and that the evaluation of distance education can be said to have moved 'beyond the packages.' Nevertheless the evaluation of the learning materials remains crucial and takes on a new significance in a dual mode situation where outside assessors need to be aware of what the material is trying to achieve in terms of facilitating learning (Rowntree, 1 99 8 ) . However, this research further demonstrates that students also wish to evaluate the whole learning transaction after feedback. In relation to performance indicators, some gaps could be directly correlated to the criteria applied across both on and off-campus modes such as assignment turnaround times and student destinations. However, when explored in detail these were shown to require more complex indicators than those applied to tradit ional on-campus graduates i n terms of more diverse motivations for studying. For instance, monitoring and evaluating the destinations of this group of students is not only a matter of quantitative analysis which can be compared across the board with various sectors but may also be of interest to see what happens to the 'products' of an innovative ILS course. In relation to this, the users of this evaluation system were also interested i n the evaluation of their student learning environment and the social contexts of their learning. The evaluation of the level of support that they are given from employers and colleagues in terms of finance, mentoring, resources, and available study time may be A R T I C L E S 1 3 valuable as evidence of the ways i n which other groups of learners within and without the LIS community are supported in developing their professional expertise through continuing education at different levels throughout their careers. In relation to this, it has been suggested that distance education needs to extend evaluation to adult learning styles (Calder, 1 994b) . For this particular group of distance learners, their learning environment is directly related to their course and as such , could be an area of research i nto constructivist ideas of learning as students identify and evaluate their experience of distance learning in relation to their working lives. Finally, the gaps identified could be seen to demonstrate how, for providers of distance education, ' the complexi ties of the administrative, supportive, and educational processes of distance education are interwoven in every part of the system' (Ljosa, 1 993, p . l 87) . This interdependence was revealed i n relation to the identification of gaps in the evaluation procedures such as the support of students at a distance, assignment turnaround, and the dissemination and utilisation of feedback, which were shown to be the responsibil ity of both academic and administrative staff. Calder ( 1 99 5 ) feels that it is important for ' learn ing organisations' to identify their predominate evaluation in order to achieve a balanced system and advocates the multiple use of collected data as one way of achieving this economically. It was clea_r that much of the data the Open Learning Unit holds about students and systems at a monitoring level could form the basis of a more informative evaluation system. The identified gaps in the evaluation procedures suggest that at present, the evaluation of the course centres on the learning materials and on the campus experience of distance learners. By considering how to extend this to the experience of students at a distance through the evaluation of admin istrative, supportive, and educational processes, the course-providers can create a more balanced, holistic, internal evaluation system that was in itself, the primary goal of this self-evaluation. References Barnett, R. ( 1 992). Improving Higher Education: Total Quality Care. B uckingham: Society for Research into Higher Education and Open University Press. Calder, J . ( 1 994a). Programme Evaluation and Quality. London: Kogan Page. ·calder, J. ( 1 994b). "Course Feedback: its Costs and Benefits; its Limitations and Potential ." In Dhanaraj an et a l . , eds . Economics of D is tance Education. Hong Kong : Open Learning Institute of Hong Kong, 24 1 -255. Calder, J. ( 1 995). "Evaluation and Self-improving Systems." In Lockwood, F. ,ed. , Open and D istance Learning Today. London : Routledge, 36 1 -369. Cresswell, R.and Hobson, P. ( 1 996). "Fallacies and Assumptions in the Use of Student Evaluation of Distance Education Teaching Materials." Distance Education, 1 7 ( 1 ) , 1 32- 1 44. Dearing, R. ( 1 997) . Higher Education in the Learning Society. National Committee of Inquiry into Higher Education, 1 997. (Led by Sir Ron Dearing). UK: HMSO. Dillon, C. and Gunawardena, C. ( 1 992). "Evaluation Research in D istance Educat io n ." Br itish Jou rnal of Educational Technology, 23 (3), 1 8 1 - 1 94. Evans, T. ( 1 994 ) . Understanding Learners in Open and D istance Education. London: Kogan Page. Fasick, A. ( 1 995) . "The Future of Education for the Information Professions." Education Libraries, 1 9 (2), 1 5 - 1 9 . Green , D.ed . , ( 1 994) . What is Quality in Higher Education? Buckingham: Society for Research into Higher Education, Open University Press. Harris, D. and Bell , C. ( 1 986). Evaluating and Assessing for Learning. London: Kogan Page. Harvey, L. ( 1 999). "The Sense in Satisfaction ." T he Times Higher Education Supplement, London, UK, January 1 5 , 29. Jakupec , V. and K i rkpatr ick , D . ( 1 997 ) . "Problems o f Generalisation i n Higher Degree Research in Distance Education" in Evans, T.et aL eds . , Research in Distance Education. Geelong, Austral ia: Deakin University Press, 4, 1 97-2 1 0. Len tell, H. ( 1 995) . "Quality : Is It Always a Move to Better Things?" in D. Sewart, ed. One World Many Voices: Quality in Open and Distance Learning. Selected Papers from the 1 7 th World Conference of the International Council for Distance Education, B irmingham, United Kingdom, 26-30 June, I n ternat ional Counci l for Di stance Educati o n . Gjerdrums vei 1 2 , N-0486 Oslo 4 , Norway, The Open University, Walton Hall, Milton Keynes, MK7 6AA, UK, (2), 1 2 1 -24. EDUCATION LIBRARIES VOL. 24, Nos. 2/3 1 4 A R T I C L E S Lippiatt, D. ( 1 997). "The Continuity of Quality Assurance in DOL and Traditional Education" in C.P. Chooi et al . , eds. Quality Assura11ce in Distance and Open Learning. Papers prepared for the I I th Annual Conference of the Asian Association of Open Un ivers ities (AAOU) , Kuala Lumpur, Malaysia, November 1 1 - 1 4 , lnstitut Teknologi MARA, 40450 Shah A lam, Selangor, Malaysia ( 1 ) , 157-62. L j osa, E. ( 1 993). "Understanding Distance Education." In Keegan, D., ed . , Theoretical Principles of Distance Education. London: Routledge, 1 75- 1 87 . Manpower Services Commission . ( 1 987) . Ensuring Quality in Open Learning: a Handbook for Action. Manpower Services Commission, Sheffield, UK. Avai lable from MSC, Room W I I I I , Moorfoot, Sheffield, S I 4PQ, UK. Morgan , A. ( 1 984)."Report on Quali tative Methodologies in Research in Distance Education." Distance Education, 5 (2), 258-266. Morgan, A. ( 1 993) . "Theorising Quality and Quality Assurance in Open and Distance Learning." In Tait, A . , ed. Conference on Issues of Quality for New Models of Education, 28-30 September 1 993, Downing CoJlege, Cambridge, UK, The Open University UK, East Anglian Region, 175- 1 83 . Morgan, A . ( 1 997) . In Evans, T. e t al, eds. Research in Distance Education, 4, Geelong, Austral ia: University Press. Patton Quinn, M. ( 1 980, 1 990). Qualitative Evaluat'on and Research Methods. London: Sage Pubs. Quality Assurance Agency for Higher Education, UK (QAA), ( 1 998) . Higher Quality, The Bulletin for the QAA, 4, (4). Quality Assurance Agency for Higher Education, Southgate Street, Gloucester, GLI I UB, UK, http://www.qaa.ac.uk Quality Assurance Agency for Higher Education, UK (QAA), ( 1 999). Guidelines on the Quality Assurance of Distance Learning, Quality Assurance Agency for Higher Education, Southgate Street, Gloucester, GLI I UB, UK. Robinson, B. ( 1 994). "Assuring Quality in Open and Distance Learning." In Lockwood, F., ed. Materials Production in Open and Distance Learning. London: Paul Chapman Pubs. Robinson, B. ( 1 995) . "Research and Pragmatism in Learner Support." In Lockwood, F., ed. Open and Distance Learning Today. London: Routledge, 22 1 -23 1 . Rowntree, D . ( 1 992). Exploring Open and Distance Learning. London: Kogan Page Rowntree, D. ( 1 998). "Assessing the Quality of Materials-Based Teaching and Learning." Open Learning, 1 3 (2), 1 2-22. EDUCATION LIBRARIES VOL. 24, Nos. 2/3 Rumble, G. ( 1 992)."The Competitive Vulnerability of Distance Teaching Universities." Open Learning, 7 (2), 3 1 -45 . SATURN . ( 1 992) . Quality Guide for Open and Distance Learning: Pilot. SATURN, Keizersgracht 756, 1 0 1 7 EZ Amsterdam, The Netherlands. Sewart, D. ( 1 993) . "Student Support Systems in Distance Education." Open Learning, 8 (3), 3 - 1 1 . Tait, A.( 1 993). "Systems, Values and Dissent: Quality Assurance for Open and Distance Learning." Distance Education, 1 4 (2), 303-3 1 4. Thorpe, M. ( 1 993). Evaluating Open & Distance Learning. UK: Longman. Unwin, L., K. Stephens et al . ( 1 998). The Role of the Library in Distance Learning: a Study of Postgraduate Students, Course Providers and Librarians in the UK. London ; New Providence, NJ: Bowker-Saur. Susan D. Lithgow is a Development Officer responsible for producing course material for library and information studies distance learning students and developing a research profile for the Open Learning Unit at the University of Wales, Aberystwyth. E-mail : sdl994@ aber.ac.uk Clare Thomas i s Director of the Open Learning Unit at the University of Wales, Aberystwyth. She leads a staff team of 10 who work to develop and support open and distance learning development, primarily for the 500+ library and information staff who are s tud y i n g for undergraduate and pos tgradu ate qualifications with DILS, and also across the Un ivers ity. E-mail: cmt@ aber.ac.uk Marianne Tay lor is a Development Officer i n the Open Learning Unit at the University of Wales, Aberystwyth, where her main responsibil ities include administration, organising study schools, student support, and study skiJls development. E-mail : mmt@aber.ac .uk The contact address for aJ I three authors is : Open Learning Unit, Department of Information and Library Studies, University of Wales Aberystwyth, Llanbadarn Fawr, Ceredigion, SY23 3AS, Wales, UK. Phone: +44 (0) 1 970-622 159 ; Fax: +44 (0) 1 970-662 1 90.