nouwens Evaluation and the development of quality learning materials Fons Nouwens and Peter Robinson University College of Central Queensland Learning materials development has traditionally been controlled by individual academics as distance education followed organisational models provided by traditional face to face teaching. Recent developments in both education and training have increased expectations of distance education. Increasing student participation rates, accessibility of higher education, increasing costs and exponential growth of knowledge are some factors that require development of innovative approaches to meet these expectations. Quality management literature suggests that these challenges may be met by a flexible but systematic, participative and team-based approach using quality improvement strategies. Developments in educational evaluation indicate that quality is promoted by an action-evaluation paradigm based on critical theory. Action evaluation promotes information gathering directed towards the making of specific decisions, a systems approach to evaluation, participative democracy in both decision making and evaluation and reflective practice. A marriage is proposed between action evaluation and quality management to guide the development of quality in distance education. Three strategies are suggested immediately: use of a team approach to materials development that is genuinely participative and democratic, collection of information about the quality of service provided to students in a way that promotes systematic improvement in the quality of that service and finally examination of all relevant aspects of educational services provided. This paper discusses the initial stages of the trial of such an approach as it is being developed within the Department of Mathematics and Computing at the University College of Central Queensland. The number of universities that the Australian Government will fund to develop materials for distance education has been reduced from around forty eight to eight. The eight institutions were given the titles Distance 94 Australian Journal of Educational Technology, 1991, 7(2) Education Centres or DECs. One goal for this rationalisation was to improve the quality of distance learning materials. If the Distance Education Centres are to meet these expectations of government, provide students with higher quality learning materials and if they are to realise opportunities presented by the growing demand in industry for training materials, they must be seen to provide better services and products than has been available in the past. Improvement of quality in distance education certainly requires establishment of standards for quality assurance, but long-term improvement in quality must also address organisational change. Literature dealing with quality improvement in the world of business and industry suggests that quality is the key to survival in a competitive environment and that institutions that wish to improve quality must introduce changes at all levels of their organisation. This paper presents an organisational model that provides a framework for discussion of the pressures for change within and external to the DECs and suggests industry experience in the field of quality as one source of strategies for organisational change for quality. A second possible approach to quality improvement in distance education, action evaluation, is then described. Action evaluation is an approach to educational evaluation that requires some adaptation to the materials development process in distance education. There is significant compatibility between quality improvement strategies and those suggested by action evaluation. Industry approaches to quality improvement and action evaluation strategies have led to recent developments that seek to improve the quality of education provided by the Department of Mathematics and Computing and the Division of Distance and Continuing Education at the University College of Central Queensland. Change: An opportunity for quality Owens' (1981) organisational model will be used as a tool to examine internal and environmental pressures that are influencing the processes of distance education. This model suggests that significant institutional change may be required to deal effectively with opportunities and threats facing distance education. Later this paper will address both quality and action evaluation strategies for dealing with these changes but first it explores Owens' model. Figure 1 suggests four primary organisational sub-systems that must be considered in assessing change in an institution. These are the tasks to be performed, the people who undertake those tasks, the structures within which they operate and the technology applied to achieve organisational ends. In Owens' model, these four subsystems should work together towards the organisation's goals and at the same time remain responsive to the environment -- the community in which the organisation exists. Nouwens and Robinson 95 Figure 1: A Model for Educational Institutions (adapted from Owens, 1981) A model for sociotechnical systems Owens (1981) applies a systems theory approach to develop a model for educational institutions. He suggests that the two central concepts of systems theory are multiple causation and the concept of subsystems. Multiple causation means that for any action there may be more than a single cause and the action is likely to be caused by a number of actions from different agents. The concept of subsystems implies the whole can be meaningfully described in terms of its parts. It does not imply that the actions of the parts are independent. Another major characteristic of this model is that the institution has an open boundary. That is, there is interaction between the institution and its environment. Some influences of this environment on the DECs will be outlined in the following section. Two further important concepts for the sociotechnical model are equilibrium and homeostasis. Equilibrium is a balance between the demands of the organisation and the needs of the individual while homeostasis is the tendency of an open system to regulate itself. The model suggests that change in the DEC environment will lead to regulatory, homeostatic processes that define a new image and new territories for the operation of each DEC. A more effective response to this new environment will be made by those DECs that can productively adapt their internal subsystems to the new environment and simultaneously develop new equilibria between organisational and individual needs. 96 Australian Journal of Educational Technology, 1991, 7(2) Distance education, especially the development of quality learning materials and the introduction of new delivery technologies, adds complexity to traditional teaching processes and requires significant change in traditional academic structures and processes to take advantage of the new opportunities available to DECs. Owens' organisational model will be used to map difficulties faced by DECs before investigating strategies for improvement. Environmental pressures for change Exploration of Owens' model begins with the environment in which distance education institutions operate. The rationalisation of the DECs and the government White Paper on Higher Education (1988) are key events in this environment. The White Paper seeks a growth of 50% in the annual output of university graduates in Australia in twelve years and identifies distance education as one means for fostering a growth in output from universities. It also sees distance education as a means of improving equity in higher education. The rationalisation of distance education was drawn by the horses of efficiency and effectiveness. The challenge to the designated DECs are to contain costs and to improve the quality of distance education making it more effective in meeting the identified national needs. Other environmental forces work to make it more difficult to meet this challenge. Staff in DECs changing from college of advanced education to university status find institutional expectations that they undertake research placing pressure on time previously allocated to the service of distance learners. In addition, as a wider range of students are recruited to tertiary studies, teaching staff find that much greater demands are being placed on their time because they encounter greater numbers of learners ill-prepared for their studies. Expectations of distance education are changing. Distance education began at many other institutions as it did at UCCQ, as a staff initiative within individual schools or departments. It grew in the early seventies as the result of a perception within individual departments of the opportunities of distance education and from the need to maintain viable student numbers. Initially numbers of students tended to be low and resources were limited. Now distance education is identified as a national initiative and is attracting resources. It is seen as a possible solution to containment of costs as participation rates in higher education rise; a source of learning resources for open learning on campus and as a way to provide accessible and ongoing education to mature learners as knowledge and careers change at a greater pace. There is a need to improve quality and performance if distance education is to meet these national needs. Nouwens and Robinson 97 Government attention to distance education and concomitant resourcing has changed the distance education environment in other ways. While cooperation between DEC and non-DEC providers is encouraged by policy and has occurred in a number of instances, competition between DECs for contract clients makes these institutions less inclined to cooperate and share resources. Federal Government rationalisation of DECs has made industry more aware of the services offered by distance education institutions and recent government training initiatives have presented new opportunities for contracts (and competition). Goals In Owens' model, organisational goals guide the expression of the institution in the environment. The distance education environment has changed significantly in recent years requiring adaptation of institutional goals to respond to the new environment. The distance education goals of the sixties and seventies that grew out of the initiatives of individuals and departments and their need to survive must be broadened to incorporate wider, national goals. If competition between DECs for the provision of services is the model for the future, the literature on quality suggests that those institutions that can offer quality services will find their future most secure. Juran (1989) suggests that improvement of quality requires institutions to include in their strategic plans specific quality goals; to allocate resources for their attainment; to generate commitment to these goals and to bring about organisational development required to achieve them. These strategic goals can be translated into more specific goals for the four organisational subsystems, that is tasks, structures, people and technology. Tasks What is the task of quality distance education? A short definition of quality is fitness for use Juran, 1989). How is distance education defined as "fit for use"? Should it be a vehicle for the content of the discipline, or should the discipline bend to the needs of learners and the needs of the community? Increasing participation rates in tertiary education are broadening the range of student ability within each class. Gender equity issues, cross- crediting, mature age student participation and the multicultural nature of student groups are just some of the environmental factors that demand a change in teaching approaches that might have succeeded with the student monoculture of the 1960s. The issue is not one of whether to maintain or lower standards, but of looking for more effective approaches to the task of teaching this disparate student group and of looking critically at the needs of each discipline in an era of change. For many 98 Australian Journal of Educational Technology, 1991, 7(2) institutions there is an urgent need to clarify the task they face, develop more effective approaches to teaching and to avoid blaming schools or students for teaching difficulties. The adaptation of teaching methods to account for a wider range of learning needs has been a major focus of secondary and tertiary education for some time. The problem faced by DECs is not essentially different from other institutions. The methods used to address the problem may need, however, to be more creative. As Figure 2 illustrates, a combination of lectures and tutorials in traditional university settings accounts for a range of learning needs and autonomy amongst learners. Many distance education courses follow a linear limited-interaction style that is similar to many large-group, face-to-face lectures which do not give learners a choice of learning paths to suit individual needs. Figure 5 shows that such a linear structure is not a necessary feature of distance learning materials. Materials with branching structures that allow learners to choose study paths appropriate to their needs can be produced. The challenge for distance education is firstly to find a model for course development that can offer a broader range of activities for learners by providing multiple learning paths. A second challenge is to provide learners with such options while containing development costs and avoiding disturbance of the homeostasis of the organisation beyond recovery. Some strategies for achieving this are discussed later in the paper. Figure 2: The potential for adapting materials to learners needs Nouwens and Robinson 99 Structures The assessment team that identified the eight DECs emphasised the need for quality assurance in distance education. In its selection of DECs the team considered the quality control mechanisms and systems established by the institution, the extent to which such mechanisms were formalised rather than at the discretion of individual academics, and the capacity of the institution to rectify faults where these were identified. An important structural problem within DECs relates to divided responsibility for quality between academic schools and distance education departments. It is difficult to introduce a systematic approach to quality assurance and quality improvement when responsibility is shared. Teaching staff claim responsibility for content and teaching. Design staff in distance education have specialist expertise in media and teaching. If deficiencies are found in materials, who is responsible for their correction? Who should evaluate the effectiveness of the materials? Distance learning materials are permanent and public. Poor materials reflect on the institution as a whole and on its individual members. How are these interests to be reconciled with concepts of academic freedom that have become attached to traditional face-to face teaching which is transient and takes place behind closed doors? The quality literature discussed later suggests a need for structural change. It could be argued that distance education, especially materials development, is so different from face-to-face teaching that the normal university structures should not apply. Materials development operates on different time lines, it should begin a year before delivery and funding is required well before delivery. In the lecture room, programs may be changed from year to year without cost or quality problems. Quality learning materials are, however, expensive to produce, and this investment in staff time and materials needs to be protected by quality improvement processes that depend on documented planning, systematic evaluation of materials and services and elimination of weaknesses in the materials. Problems occur when no structures exist to protect this investment. People Rationalisation of distance education and industry training needs have generated significant work for both DEC and non-DEC providers and there is an increasing need for specialist expertise to prepare quality learning materials. In addition, pressure for development of new delivery technologies is leading to demands skills in these areas. Often lecturers employed for their academic skills and many staff employed for their expertise with media are new to distance education. Systematic approaches are required both for induction of new staff and for the 100 Australian Journal of Educational Technology, 1991, 7(2) development of new skills required by staff at a time of growth and change. Imagination and an empathy for the learner are required in distance teaching because such teaching is expressed in ways that are very different from those employed in the classroom. This empathy is expressed in many ways; style of written language, interesting materials, organisation of the materials, anticipation of problems and attention to details like choice of text and checking of text availability. All the skills required to prepare quality materials rarely reside in one person. Thus another challenge within the people subsystem is to encourage effective teamwork among writers, design, production and delivery staff. Materials development for distance education is a different exercise from on campus teaching. A face-to-face lecture can be planned quickly a day before delivery, and problems dealt with during the lecture. In the distance mode, however, a problem caused by one team member can require attention of other academics, editors, designers, students and student support staff. It is important to identify and eliminate possible faults as early as possible and teamwork and planning can help achieve this. Lessons can be learned from the 'right first time' approach to quality assurance (Price 1984). Consistent quality cannot be obtained and quality improvement is difficult to implement if individual writers refuse advice and assistance of other staff affected by their decisions. The design and development of quality learning material does not follow a linear process through writer, editor, typesetter and proofreader, it is a team responsibility (Gough 1984). The publishing industry acknowledges problems in this area: "A solution to the management problem would require radically new management structures in the publishing industry. Specialist boundaries may have to be redrawn and a team approach encouraged." Waller (1977 p.150) The outcome of such an approach is seen in Open University materials, popular handbooks like Time-Life and Readers Digest publications and modern textbooks like Microeconomics (Parkin 1989) that contain many of the features of good distance learning materials. The second part of this paper draws from quality improvement literature and action evaluation processes to develop strategies for dealing with these problems. Technology Improved quality and availability of communication technology have expanded possibilities for development and delivery of distance education courses. The opportunities this technology presents must not be misused. The Dawkins White Paper (1988) emphasised the need to apply advanced technologies to distance education and the need to ensure the broadest possible access to geographically isolated areas. This presents a dilemma because currently the most reliable and cheapest access to isolated areas is by post and telephone while many new technologies advocated for Nouwens and Robinson 101 distance education cost more to transmit and require a large investment in hardware for individual students. Indeed for many institutions, promotion of 'advanced technologies' suggests more use of electronic publishing, teleconferencing and the mailing of audiotapes, video tapes and computer disks. Limited resources for materials development restrain the effective use of these technologies in some subjects and allow continued, inappropriate use of these technologies in other subjects. A systematic approach to subject development is necessary to ensure that appropriate use is made or technology, that it is used to deal with teaching and learning needs, that its use is justified when compared with other options, and that it is integrated with the study package and not used for its own sake. Use of quality learning materials, possibly mediated by computer managed learning systems, may provide a platform for resource based learning, both on and off campus and in industry. Such systems have potential to individualise learning and may lead to more effective use of academic staff time. Owens' model suggests such changes in one organisational subsystem, for example, technology will lead to a change in the equilibrium between the institution and individuals within its other subsystems. The previous discussion has outlined internal and external pressures for institutional change and the need to manage organisational change, to take advantage of opportunities presented and to achieve quality in distance education. The following sections suggest strategies for introducing such changes. Applying quality in distance education This section will introduce concepts dealt with in literature on quality in industry. This literature suggests possible strategies for dealing with organisational change in a way that promotes improvement of quality in distance education. Application of these concepts will be taken up later when the introduction of structures and procedures for quality improvement between the Department of Mathematics and the Division of Distance and Continuing Education at UCCQ is discussed. Much of the quality approach in industry has its origins in the need of Japanese industry to overcome a pre-war reputation for poor quality in order to become competitive in world markets. The Japanese success in industry has been partly attributed (Juran 1989) to a focus on quality, and this success leads to an understandable desire to incorporate their practices in Australian organisations. Holmes (1981) discusses a tendency amongst educators to look for easy solutions by cultural borrowing from other societies. Uncritical cultural borrowing is an example of what Vandenberg (1985) refers to as technicism: "technology used destructively 102 Australian Journal of Educational Technology, 1991, 7(2) or in a dehumanizing way, or too efficiently, or automatically when some other approach might be more appropriate". Application of an industrial solution, particularly one based on the success of another culture, may be optimistic at best and dehumanising at worst. Holmes (1981 p.340) suggests a more reflective approach is necessary when borrowing strategies for change, "The successful planning of educational development depends upon the care with which we refine techniques and models to describe local needs and conditions and to formulate generalisations from which predictions can be made". Implementation of Asian organisational principles should be cautious and take full account of the cultural differences. Miller (1984) identifies three basic types of organisations: traditional American corporations (Type A), Japanese organisations (Type J) and a hybrid of the two (Type Z) . He proposes that two particular aspects of Type Z philosophy should have immediate application in educational settings: the development of a philosophy and a cooperative team approach. Table 1: Contrasting types of organisations (Miller, 1984) Type A (American) Type J (Japanese) Type Z (Hybrid) 1 Short term employment 1 Lifetime employment 1 Long-term employment 2 Individual decision making 2 Consensual decision making 2 Consensual decision making 3 Individual responsibility 3 Collective responsibility 3 Individual responsibility 4 Rapid evaluation and promotion 4 Slow evaluation and promotion 4 Slow evaluation and promotion 5 Explicit formalised control 5 Implicit formal control 5 Implicit formal control; explicit, formal measures 6 Specialised career path 6 Non-specialised career path 6 Moderately specialised career path 7 Segmented concern 7 Holistic concern 7 Holistic concern for individuals Miller (1984) identifies the benefits of adopting a Type Z organisational philosophy as: • increased work satisfaction, • more cooperative relationships at work and in personal life, • greater motivation and enthusiasm, • better communication, • increased awareness and acceptance of growth and change. Organisational changes of this type have not been eagerly adopted by educational organisations. This may be due to lack of knowledge by decision makers, inertia, or because of concerns about the costs of quality. Nouwens and Robinson 103 Outside the education arena, many organisations are examining the costs associated with not producing quality products. A consensus is developing in both goods and services industries that survival in any competitive environment depends on the development of a quality consciousness at all levels of organisations. Figure 3 was adapted from Juran (1989) to show how this quality consciousness can produce results that go beyond the 'quality control' models that previously guided industry. The traditional 'quality control' approach (Figure 3(a)) shows a concern with controlling the cost of poor quality; the goal is to control the work process to meet defined standards. The quality management model promotes quality improvement as the key to survival in a changing world. This approach requires the development of different relationships between the people working in an organisation. Figure 3: Systematic quality improvement 104 Australian Journal of Educational Technology, 1991, 7(2) Management for Quality Keeping in mind the previous cautions about cultural borrowing, the literature on quality improvement in industry is one source of strategies for organisational change within DECs. Juran (1989) suggests that a strategic, institution-wide commitment is required to produce lasting improvements in quality. He presents three processes as the basis for managing for quality: quality planning, quality control and quality improvement. Juran identifies quality improvement as the first process that should be addressed because it provides an earlier and more measurable return. The quality improvement process consists of four steps. 1. Establish the infrastructure needed to secure annual quality improvement. 2. Identify specific needs for improvement, that is, improvement projects. 3. Establish a project team with responsibility for completing the project. 4. Provide the resources, motivation and training needed to diagnose the cause, establish a remedy and establish controls to hold quality gains. The aim of quality improvement is the organised creation of beneficial change. It is directed at quality with a 'Big Q', that is at improving all services; all products for internal and external use; all processes used in the organisation and all organisational relationships, internal and external. It is directed at improving products and services as well as at elimination of defects. A key concept in quality management is that the organisation, its subsystems and the individuals within it commit themselves to service of customers (Saunders, 1991; Juran, 1989). This commitment to customers provides a measure that assists dispute resolution and helps the organisation to focus decision making on its goals rather than on internal issues. Initiation of change for quality requires an authoritative forum to generate commitment, allocate resources to quality improvement, to set goals, provide policies, provide staff development and to allocate responsibilities. A quality committee may provide this forum and deal with such matters as developing a project nomination and selection process, selecting project teams, preparing teams, developing measures for quality, supporting projects and providing recognition to staff involved. An important feature of the approach advocated is a strong commitment from senior management to provide resources for quality improvement and for project teams, and to spend a significant part of their own time dealing with quality issues. Senior staff must be seen to be committed to identifying problems and providing resources for their solution. Nouwens and Robinson 105 Quality improvement is a project-by-project exercise. Each project deals with one problem at a time and each project is undertaken by a project team. Nominations for projects should be solicited from all staff at all levels and from outside the organisation (eg. students, government, industry). Criteria for project selection should be defined before selection is made, and care is needed selecting early projects to ensure they are feasible and significant, have measurable results and address a chronic problem. The quality committee selects projects to be resourced by applying the defined criteria. Owens' people subsystem is addressed by a cooperative approach also advocated by Miller (1984) in his Type Z organisation. Juran (1989) suggests that a project team of about seven members be chosen to include representatives of all departments involved, people with the required expertise and people with time and inclination to work on the project. Teams usually meet weekly and report regularly to the committee. The major focus of project teams is systematic and incremental improvement. Strengths and weaknesses of the service or product are identified and effort is concentrated on eliminating errors and overcoming weaknesses one at a time. This evolutionary approach reduces the risk of introducing new faults to the features of a service or product with which the customer is satisfied and seeks to justify all changes by collection of information and discussion within the team. Application of this approach to the subject development process is illustrated in Figure 4. After an initial offering the major weaknesses of a subject are identified and prioritised given the resources available. Weaknesses may be addressed in a number of ways, for example, by revising materials or by supporting the learner with additional media. This means that the course of instruction may initially be presented as a traditional, linear learning package as illustrated in Figure 5(a). Additional materials employing a variety of media are then developed to increase the range of learning needs satisfied by the course without destroying its basic integrity or making unnecessary changes to the materials already developed. Over time a learner needs package is developed which may have a structure illustrated in Figure 5(b). 106 Australian Journal of Educational Technology, 1991, 7(2) Figure 4: Systematic quality improvement Figure 5: Systematic development of options for learners !using quality improvement processes Typically it takes several years to establish quality improvement as an integral process in an organisation. This quality improvement process requires a large change in the culture of most organisations. The literature (Juran 1989; Price, 1984; Saunders 1991) suggests other useful strategies for quality improvement as well as supporting quality assurance processes designed to maintain the gains made in the quality improvement projects. Nouwens and Robinson 107 The basic quality concepts fit well with Owens (1981) model. Saunders (1991) used a model similar to Owens to analyse problems with quality. The quality approach follows Miller (1984) in suggesting a cooperative team-based approach and the development of an organisational philosophy that is client centred. It is also supported in many of its strategies, for example the Deming cycle (Saunders 1991) and by the action evaluation process that will now be described. Evaluation: Some theoretical considerations The strategies suggested by the quality approach in industry have at their base a process of evaluation and the setting of standards for evaluation. Adoption of these strategies would seek to apply the process of evaluation to all aspects of distance education, including distance teaching, an area in which there is only a very limited tradition of formal evaluation. In entering this arena, clear concepts regarding the nature of evaluation are required to encourage productive debate. Epistemology of evaluation Codd (1988 p.6) discusses the epistemological traditions underpinning evaluation theory. He suggests: "In educational evaluation, theories of knowledge have an important bearing on what is evaluated and on the activity of evaluation itself." He describes two antagonistic traditions or frameworks, positivism and hermeneutics, into which approaches to evaluation can be categorised. The assumptions of positivist evaluation can be summarised as: • The only knowledge that we can have of the real world comes from sense experience. Scientific knowledge is the result of direct observation of perceived things and processes. • For every genuine question, there is only one true answer, and for every problem, there is only one best solution. • There is only one rational method of solving problems. That method is based on universal agreement over the law-governed nature of reality and the predictability of events. • Scientific knowledge is entirely factual, cumulative and progressive. (Codd p.10) The main aim of positivist evaluation according to Codd (p.12) is: "to make available to administrators an information base for use in choosing the 'best means' of achieving certain predetermined goals." This top-down approach has been found wanting in quality improvement in industry. It also reflects the processes used in traditional instructional design systems whose origins lay in military training and which are based on a linear, prescriptive approach to learning materials development. The adoption of this positivist approach is criticised by Codd on the grounds that it is 108 Australian Journal of Educational Technology, 1991, 7(2) essentially an economic view of education. Evaluation under this model treats human satisfaction in the same way as the efficiency and effectiveness of a program, that is, something to be optimised. The presumed neutrality of this positivist approach drives a wedge between theory and practice, fact and value, means and end. These dichotomies leave assumed values unexamined and existing power structures unquestioned. They state in bold terms what are in practice tentative assumptions about knowledge and overstate the positivist's confidence in the results obtained. For example in the statement " scientific knowledge is entirely factual, cumulative and progressive" Codd (p.10) has created a positivist model which is easily criticised. This model, however, is not likely to be the model adopted by most positivists. Indeed any criticism of the positivist tradition must acknowledge the spectacular success of positivism in creating modern industrialised society (which few would wish removed in its entirety). The essential strength (and also main problem) with positivism is that it makes conservative predictions about those phenomena which can be described adequately by scientific models. To reject the methodologies of positivism is to throw the baby out with the bath water for even within a liberated society without the manipulation of power elites it is likely that the aims of efficiency and effectiveness and the utilitarian aim of greatest happiness for the greatest number would still be valued. Codd himself (p.28) recognises that the issue is not the methodology of positivism. Codd (1988) also describes the interpretive (hermeneutic) tradition of educational research. The assumptions of this tradition, according to Codd, are that all perception is theory-dependent and that human action can be understood only within particular contexts. The central aim of this research tradition is "examine such things as what a situation means to those individuals involved in it, what their outlook is, and what frames of reference they use to make sense of the world." (Codd p.17) Codd explains that while evaluation strategies based on this interpretive tradition have overcome some of the manipulative and oppressive features of positivist evaluation their focus on values maintains the fact/value dichotomy. As a result of its tendency towards cultural relativism, this tradition has "remained essentially passive in the face of prevailing social relations" (Codd p.21). In the field of quality in organisations, this approach is problematic. It suggests a collection of autonomous individuals making decisions based on their individual perceptions of quality. It takes no account of the complexity of the skills involved in developing learning materials and the need to coordinate effectively skills which are distributed in the workplace. Such an approach occurs in most academic institutions. Academic freedom, a critical atmosphere, the individualising nature of the lecturing task, isolation by timetabling, departmental autonomy and the competitiveness of the research process lead to organisational climate and structures that promote autonomy. Such an Nouwens and Robinson 109 internal environment encourages interpretative evaluation strategies that make quality improvement difficult to implement. To overcome the conservatism of both the positivist and hermeneutic traditions of evaluation, Codd advocates a Critical Theory approach based largely on the work of Jurgen Habermass and the Frankfurt School. Kemmis (1984) discusses the distinction between traditional hermeneutics and critical social science. Critical social science applied to education is committed to the improvement of the quality of education and uses strategies similar to those advocated in industry (plan/act cycles, teamwork, project focus, sharing of power, evolutionary change and decision-making (Juran 1989)). According to Kemmis, If research is to achieve concrete transformation of real educational situations, then this requires a theory of change which links researchers and practitioners in a common task in which the duality of the research and practice roles is transcended. It requires joint participation and collaboration in the process of social transformation, expressed in joint participation in the decision-making process of transformation.(p.31). Critical theory rejects the assumptions about objective knowledge of positivists while maintaining the conviction that all social science should be empirically grounded. It also seeks to "develop interpretive contextual categories, while recognizing that many of the actions people perform are influenced by social conditions and constrained by structures over which they may have little or no control"(Codd p.28). Critical theory recognises the centrality of moral values and entails a commitment to action. An acceptance of the critical theory approach means that evaluation needs to concentrate on improving practice by making choices that are based on reflection and evaluation of reasonable, available choices. According to Habermass (1971) it also means taking a systems view of educational practice - in short, examining all aspects of practice to ensure that all alternatives are explored and that all parts of the structure are subject to critical reflection. A systematic approach, but not the linear positivist approach, is necessary to ensure that the materials development processes in distance education lead to the examination of all aspects of practice. Action evaluation provides a basis for such an approach. Action evaluation Batchelor and Maxwell (1987) describe the action evaluation approach which is based on critical theory. They define action evaluation to be: a process in which the 'practitioners' are included as evaluators, which features collaborative planning and data-gathering, self-reflection and responsiveness, and which embodies a substantial element of professional development. 'Ownership' of the evaluation is vested in the 'practitioners' (p.70). 110 Australian Journal of Educational Technology, 1991, 7(2) Action evaluation recognises the importance of the practitioner's commitment to change. It stresses professional development and critical self reflection. Data-gathering focuses on improvement and methods vary depending on particular decisions and their context. Action evaluation involves cooperation within teams through a consensus-building, democratic process. Other strategies include an iterative, quality improvement process: • Adoption of an evolutionary rather than revolutionary approach to decision-making and information-gathering. • Identification and elimination of major weaknesses in programs, especially those that are readily discernible. • Identification of strengths of the program which have been reviewed or are obviously not in need of review to protect them from unnecessary change. Some limitations to be considered in implementing action evaluation are as follows. Firstly, unless resources are available, information-gathering methodologies are limited to those which are practicable within a short time frame. Secondly, the knowledge gained is unlikely to be generalisable to other departments and will probably be applicable for only a limited time-span. Thirdly, most proposed actions will be those perceived as achievable within existing resource and institutional constraints. While solutions can be imaginative, the model is evolutionary and it is unlikely that practitioners will attempt to force major institutional change in the short term. As well as acknowledging the limitations above, the following assumptions are made: • Sufficient information is available within the institution without the need for access to outside consultants. • Academic and non-academic staff are able to identify the real needs of the students. • Students are encouraged to reflect on their situation with sufficient insight to allow programs to be modified based on the information they provide. Quality and action evaluation The approaches to quality in industry and action evaluation in education have different origins but employ what is in many ways, similar methodology. Links between the two approaches have been identified and are summarised here. There is a client focus. They both employ iterative, evolutionary strategies to improve quality. Teams comprise representatives from all work groups that will be affected by the evaluation and subsequent action. In each case the responsibility for improvement rests with the members of the team collectively. Democratic, Nouwens and Robinson 111 consensus-building processes are employed to set reasonable goals and to determine ways of achieving them. The projects chosen may be directed at processes, products and services to both internal and external clients. The information required is collected by the team in the most convenient way and is 'owned' by the team. Both approaches follow a team building staff development model. It is in the project teams' interest to document the process sufficiently well that they not only have the information for decisions but can provide their institution or government agencies with information to ensure support and to obtain resources improvements. Adoption of a quality framework for action-evaluation allows practitioners to define collectively what is meant by quality education, to set evaluation criteria and strategies and empirically test assumptions progressively develop guidelines that will allow practitioners to deal with complexity and change. This approach also recognises the essential openness of educational organisations and the information needs of the community at large. This marriage overcomes the individual limitations of the positivist and hermeneutic traditions outlined earlier. Crucial to both approaches and therefore to the marriage is the integration of information-gathering, decision-making, and action within an action evaluation plan. The Deming cycle (Saunders, 1991) illustrated in Figure 6 demonstrates how this integration is achieved. Traditional end-point evaluation and quality assurance methods do not ensure quality improvement because the information needs of decision-makers are not met at an appropriate time. Figure 6: Quality improvement / action-evaluation cycle 112 Australian Journal of Educational Technology, 1991, 7(2) Evaluation strategy for an academic department In the past it was generally the case that individual lecturers were responsible for most aspects of subject development. In selected subjects, a course development officer was assigned to assist writers. There was, however, little formal or systematic planning, documentation or effective evaluation. Where evaluation did occur it consisted of an informal record of student problems or a short student questionnaire. Major weaknesses in this evaluation process are discussed below. Evaluation questionnaires administered to students on completion of a subject have been used with varying degrees of success in face to face teaching. Such questionnaires provide some information but used alone, they are inadequate for systematic evaluation of distance learning materials. The main reason is because not all students require quality materials, indeed a significant number of students would succeed regardless of the quality of the materials. In addition for reasons beyond the control of the teacher, some students would not achieve subject objectives no matter what teaching strategies are used. There is however a third group of students for whom the quality of learning materials is critical. The traditional data gathering strategy does not distinguish the three groups of students. It does not obtain information about the specific target groups for whom quality materials are important. Compounding the problem is the fact that the instrument is usually administered at the end of the course with very low response rates. It is likely that the small number who do respond will be those who would have succeeded regardless of the quality of the educational service, thus the information they provide may not be a useful guide to where the subject should be improved. Add to this the fact that few traditional evaluation instruments actually ask for specific areas needing improvement. They tend instead to measure the general appeal of the materials. The conclusion that this approach to evaluation of learning materials should be abandoned is inescapable. What is required is an alternative approach which focuses on information needs of decision-makers within the whole process of course planning, development and implementation. Such an approach should make maximum use of information which is already available within the organisation and employ convenient methods for collecting only information which will be used to support decision-making. In both the Department of Mathematics and Computing and in the Division of Distance and Continuing Education, staff have been concerned to improve the quality of distance education and improve teaching effectiveness. Liaison between departments has for some time followed a consultative, project-by-project style to the resolution of problems. Nouwens and Robinson 113 Building on this relationship, a Materials Development Group with membership from both departments was established early in 1991 to promote quality improvement in the development of learning materials. A priority for the group was to initiate the development of evaluation processes that would encourage reflective practice in materials development and provide mechanisms for promoting effective use of department resources in a way that supported rather than threatened those contributing to the materials. To overcome the problems listed above, the Materials Development Group is developing an evolutionary approach to implementation of an evaluation strategy based on the quality management strategies and action research. Elements of this strategy include the following. • The focus on the client (learner) suggested by the quality literature as a way of reducing conflict within organisations is a strategy that the Materials Development Group seeks to use to enhance relationships between members of course teams. • The development of a feedback newsletter reporting to students the decisions made on the basis of information collected in subject evaluation. The goal of the newsletter is to encourage students to provide more effective feedback about the quality of products and services. • The Materials Development Group seeks to develop an authoritative forum to generate commitment to quality improvement, to set goals, define responsibilities and allocate resources to materials development projects. • Provision of staff development to support quality improvement projects. • The establishment of course teams in those subject development projects where the greatest quality gains could be made by students and by staff. • A course development officer to act as a facilitator for planning, documentation and evaluation strategies and provision of editorial support. • Further development of author's kits to assist course teams to follow a systematic approach to course development. These contain instruments to document the information needs of each stage of the planning, development and quality improvement processes and will themselves be subject to evaluation and quality improvement. • The course development officer has been given the task of analysing the results of evaluation forms and analysing existing achievement data in the student records system. • Course teams have been encouraged to submit their draft materials to development testing. This involves selected students working through the learning materials at the second proof stage. 114 Australian Journal of Educational Technology, 1991, 7(2) • Course teams are encouraged to engage an external auditor to review draft materials. • DDCE is developing methods of recording and storing information received from students in the form of comments or complaints on the telephone. This is a pilot venture with the School and DDCE to test and develop procedures for quality improvement in distance education. These initiatives are relatively new and changes are slow because they follow an evolutionary model and depend on development of consensus within a quality assurance framework. Industry experience (Juran 1989) suggests that three years is required for most organisations to orient themselves fully to quality improvement processes, but that important gains can be made in the first year. Conclusion The special contribution of 'quality in industry' literature lies in strategies it offers to guide the introduction and development of systems to improve quality and to guide institutions in dealing with change. Institutions must change in order to respond to external pressures and changes will occur as those internal organisational sub-systems; people, tasks, technology and structures find new patterns of equilibrium in the interaction between the organisation and its members. Action evaluation on the other hand, provides a theoretical, education-based model to guide practitioners within institutions as they develop evaluation/decision-making processes to improve the quality of learning materials. Action evaluation supports the quality improvement literature in that both approaches adopt project and team-based, participative, consensus processes. They seek to encourage effective adaptation to change and suggest a need for organisational processes that deal with complexity and facilitate contributions to quality from all members of the organisation. Both approaches lend themselves to the development of learning materials where educational products and processes are permanent and public where incremental improvement in quality can be made by removing weaknesses and retaining strengths in materials and services. The attention paid to distance education as a solution to challenges facing higher education and the demand for resource-based training in industry points to a need to recognise the differences between development of learning materials and the processes of face to face teaching and training. Effective development of quality learning materials requires new patterns of organisation that acknowledge the contributions of all involved in producing such materials and provide supporting tools and processes. The developments that are taking place at UCCQ are initial attempts at addressing quality improvement. Effective evaluation processes are seen as the key to such improvement. Nouwens and Robinson 115 References Barrow, R. (1983). Moral philosophy for education. Unwin Education Books, London. Batchelor, M. and Maxwell, T. (1987). 'Action evaluation' new education. International Journal of Educational Theory and Practise, Vol 9, No 1-2. Beaudoin, M. (1990). The instructors' changing role in distance education. The American Journal of Distance Education, 4(2). Codd, J. A. (1988). Knowledge and control in the evaluation of educational organisations. Deakin University Press, Geelong. Dawkins, J. S. (1988). Higher education - a policy statement. Australian Government Printer, Canberra. Department of Commercial and Industrial Development (1976). Product design and development. Small Business Series, Vol 3. Foggo, T. (1988). The introduction of open learning as a change process. Open Learning, November. Gough, E. (1984). Towards a philosophy of distance education. In K. Smith (ed), Diversity down under. Darling Downs Institute Press, Toowoomba. Habermass, J. (1971). Toward a rational society. Heinemann, London. Holmes, B. (1981). Comparative education: some considerations of method. George Allen and Unwin, London. Juran, J. M. (1989). Juran on leadership for quality. The Free Press, New York. Kemmis, S. (1984). Educational research is research for education. Australian Educational Researcher, 11(1), March. Kemmis, S. (1985). Programme evaluation in curriculum development and innovation. In House, E. R. (ed), New directions in educational evaluation. The Falmer Press, London. Lewis, P. (1990). Images of chemical education: Some international perspectives. SMEC Seminar 179, Curtin University of Technology, Perth. Lindauer, I. E. (1988). Thinking skills and their significance in the public schools. SMEC Seminar 158, Curtin University of Technology, Perth. Miller, W. C. and Sparks, D. (1984). Theory Z: the promise for US schools. The Educational Forum, Vol 49, No 1, Fall. Nation, D. (1990). Reporting research in distance education. In revised papers, T. Evans (ed), Research in distance education 1. Institute of Distance Education, Geelong. Nevo, D. (1985). Conceptualization of educational evaluation: an analytical review of the literature. In House, E. R. (ed), New directions in educational evaluation. The Falmer Press, London. Nielsen, L. and Turner, S. D. (1983). Program evaluation as evolutionary process. Evaluation Review, Vol 3, No 3. Owens, R. G. (1981). Organizational behavior in education. Prentice Hall, Englewood Cliffs. Parkin, M. (1989). Microeconomics. Addison-Wesley. Price, F. (1984). Right first time. Gower Publishing Company, Aldershot. 116 Australian Journal of Educational Technology, 1991, 7(2) Saunders, I. (1991). Quality in tertiary education workshop. Department of Mathematics and Computing, University College of Central Queensland, 2 July. Thornton, H. P. (1987). Why you need the industrial designer. Machine Design, June 11. Tobin, K. and Fraser, B. J. (1987). Exemplary practice in science and mathematics education. Science and Mathematics Education Centre, Curtin University of Technology, Perth. Vandenberg, D. (1985). Ideology and education in advanced industrial society lecture series. Division of External Studies, University of Queensland, St Lucia. Watson-Guptill Publications (1988). Xerox publishing standards. New York. Authors: Fons Nouwens is Head of Instructional Design in the Division of Distance and Continuing Education at the University College of Central Queensland. Peter Robinson is a Materials Designer and Lecturer in Mathematics in the Division of Distance and Continuing Education, University College of Central Queensland. Please cite as: Nouwens, F. and Robinson, P. (1991). Evaluation and the development of quality learning materials. Australian Journal of Educational Technology, 7(2), 93-116. http://www.ascilite.org.au/ajet/ajet7/nouwens.html