25 Departmental evaluation: Speech therapy and audiology, An academic department aimed at teaching a profession S R Hugo Department of Communication Pathology University of Pretoria W Μ J Hugo Director: Graduate School of Business Leadership, UNISA ABSTRACT In a recently established policy guideline of the Department of National Education it was clearly indicated that programmes and departments at tertiary education institutions should institute evaluation procedures as a matter of urgency. This has become necessary to provide a basis for decision making and policy formation, to accredit programmes, to monitor expenditure of public funds and to improve educational material. It is the aim of this paper to develop an evaluation model, based on accepted education principles, for evaluating the actual performance of a Department of Speech Therapy and Audiology. A basic management model is used and specific functional indications are given of the steps that should be taken in evaluating a department. The model is a comprehensive one which can easily be adapted for use in other departments. OPSOMMING In 'n belangwekkende publikasie van die Departement van Nasionale Opvoeding is aangedui dat dit tans gebiedend noodsaaklik geword het dat departemente verbonde aan tersiire opleidingsinrigtings dringend moet beplan om evalueringsprogramme te loots. Die redes hiervoor is dat evaluering basisinligting verskafvir beleidsbesluite, akkreditering van programme moontlik maak, besteding van fondse monitor en akademiese opleidingsmateriaal en -stelsels verbeter. Dit is die doel van hierdie artikel om 'n evalueringsmodel, gegrond op erkende opvoedkundige beginsels, te ontwikkel virgebruik in die bepaling van waarde van 'n departement vir spraakheelkunde en oudiologie. 'n Bestuursmodel is geselekteer en spesifieke aanduidings is verskafvan die strategies, stappe en norms wat gebruik kan word in departementele evaluering. Die model is 'n omvattende een wat suksesvol aangepas kan word vir gebruik deur ander departemente. "Due to the complex nature of human communication and its disorders, it is essential that information of an academic, research and scientific nature is continually developed and evaluated. A university provides the optimal environment for constant academic validation of current professional training." Feldman 1981, ρ 942 1 INTRODUCTION Evaluation, or the need to determine the worth and merit of whatever is being evaluated, is a well-known concept. There is ample evidence that evaluation in edu- cation was an accepted practice even prior to 1920. From the literature, it is equally clear that the debate is still raging on how evaluation should be conducted and what role it should fulfil in different spheres. The emergence of educational evaluation in terms of student outcomes and institutional effectiveness has been a significant trend in higher education in the past few years (Nichols, 1989, ρ vii). In the United States of America, for example, standards are established and en- forced to ensure institutional quality. To maintain accred- ited status, institutions must be evaluated periodically to demonstrate achievement of standards and to document ongoing efforts for institutional improvement (Middleton et al., 1989, ρ 15). In Europe too, there is growing empha- sis on the philosophy of value for money, and funding for higher education is therefore being linked directly to per- formance (Goedebuure et al., 1990, ρ 15). Institutions for higher education throughout the world increasingly ac- cept the fact that they are accountable to political authori- ties and to the communities they serve. Establishing evalu- ation programmes at institutional, departmental and pro- gramme levels in higher education serves the dual pur- pose of proving quality of performance to stakeholders, while at the same time protecting the traditional autonomy of tertiary education institutions. In South Africa, education authorities clearly indicated in a policy statement that academic departments are ex- pected to institute evaluation procedures aimed at inter- nal evaluation and to plan for external evaluation accord- ing to prescribed guidelines (NASOP 02-129 87/10). In spite of the contemporary urgency of evaluation at tertiary level, no formal and generally acceptable evalua- e Suid-Afrikaanse Tydskrif vir Kommunikasieafwykings, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) 26 S.R. Hugo & W.M.J. Hugo tion model has been developed that could be used to pro- vide quantitative information on the actual performance of departments in general. Wolf (1990, ρ 19) reports that there are no less than fourteen different evaluation models and five different ap- proaches. There is also little agreement or data to sup- port the efficacy of one model or approach over another. Evidently the fundamental problem is that the various models are built on differing and often conflicting concep- tions and definitions of evaluation (Worthen & Sanders 1987, ρ 44). In view of the above, the following problem statement can be formulated: What are the principles involved in structuring an evaluation model for an academic department engaged in a professional programme aimed at developing Speech Therapists and Audiologists? 2 RESEARCH OBJECTIVES AND DESIGN The objective of the study is to develop an evalua- tion model, based on accepted educational principles, for evaluating the actual performance of a department of Speech Therapy and Audiology in South Africa, in order to address the specific needs of this unique and divers context. To attain this objective, a research plan comprising of two consecutive stages was devised: • establishing the underlying principles of evaluating programmes and departments in tertiary education institutions by means of a comprehensive literature study • developing a proposed model for the evaluation of a department of Speech Therapy and Audiology. 3 CONCEPTUAL AND CONTEXTUAL ISSUES OF EVALUATION 3.1 Terminology and concepts As previously indicated, various approaches and defi- nitions of evaluation may lead to fundamental differences in educational evaluation. The term evaluation for exam- ple, may be interpreted to mean measurement, appraisal, assessment, professional judgment, scientific enquiry, a political activity or an act of collecting and providing in- formation (Worthen & Sanders 1987, ρ 22). In this study, evaluation will be interpreted as the act of rendering judg- ment to determine worth or merit that is: ... the making of judgments about the worth and ef- fectiveness of educational intentions, processes and outcomes; about the relationships between these; and about the resource, planning and implementation frameworks for such ventures. (Adelman & Alexan- der, 1982, ρ 5.) Because of differences in interpretation and the di- vergent μββ of evaluation processes in various institu- tions and also in different countries, the meaning at- tached to evaluation concepts and procedures may be utterly confusing. Nisbet (1988, ρ 4) suggests a cogni- tive map to assist in establishing a common understand- ing of concepts and their role in evaluation. This is rep- resented in Figure 1. Only the most important (and per- haps most controversial) concepts referred to in this fig- ure will be dealt with here. The controversy about the formative or summative na- ture of evaluation has raged for many decades. In essence, the question is whether evaluation should provide imme- diate feedback of information useful to programme devel- opers who would use the information for revision during the development process (formative evaluation). Contrary to this point of view, summative evaluation is conducted at the end of the programme and is aimed at providing judgment about the value of a programme. Since the evalu- ation of an academic department is the subject of this study and because academic departments, like academic pro- grammes are forever changing and in a state of develop- ment, formative evaluation is of crucial importance. How- ever, it is equally important to provide feedback to au- thorities about the actual performance of an academic department at particular times (at the end of an academic or financial year or review cycle, for example). Hence evalu- ation in the context of this study must also perform a summative role. Linked to the issue of formative and summative evalu- ation is the question of internal or external evaluation. Because of the immediateness of formative evaluation, in- formation provided by this type of evaluation is valuable to departmental managers and often used to effect im- provements. This type of evaluation is therefore frequently undertaken internally with internal evaluation objectives in mind. Summative evaluation, however, can be used for accountability and is often undertaken by external evalu- ators commissioned by some external audience or deci- sion maker. As an extension of the argument that formative as well as summative evaluation play an essential role in depart- mental evaluation, it is evident that in evaluating the aca- demic department, both internal and external evaluation are essential. The quantitative or qualitative base of evaluation is another conceptual issue referred to in the cognitive map depicted in Figure 1. Qualitative inquiry focuses on proc- esses rather than outcomes and uses an inductive ap- proach to data analysis while quantitative inquiry focuses on numbers by emphasizing standardization, precision, and objectivity (Worthen & Sanders, 1987, ρ 51). j However, in departmental evaluation quantitative and qualitative methods can be regarded as compatible and complementary approaches. Worthen and Sanders (1987, ρ 53) support this premise and point out that the trend in evaluation today is towards methodological pluralism. The concepts of accountability and professionalism are of particular importance in the context of evaluation (see Figure 1). Educational accountability is the responsibil- ity for identifying and measuring educational outcomes and using the information about these outcomes for deci- sion making (Kogan, 1986, ρ 75). Accountability has three major dimensions in education. Moral accountability is related to the fact that academic institutions are answer- able to their clients. This implies that if resources are spent on educational programmes, it is necessary to determine whether the programmes (or departments) achieve what they are designed to achieve (Lee & Sampson, 1990, ρ 157). Evaluation of performance outcomes provides proof of aca- demic quality and it is important to note that funding for higher education is increasingly linked to the quality of this performance (Goedegebuure et al., 1990: 15). Profes- sional accountability refers to the fact that academicians are answerable to themselves and to their colleagues in The South African Journal of Communication Disorders, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) Departmental Evaluation: Speech Therapy and Audiology 27 Η Top-down Hard data Appraisal External evaluation Outward-looking Accountability Teacher as trained practitioner Assessment Testing Control Performance indicators Product Summative Judgement EVALUATION η % % % Formative Flexibility Process Understanding Insights Growth Management Objectives Planning Rational X Teacher as reflective professional % Teacher involvement Commitment Consensus building Internal evaluation Self-evaluation Grass-roots Professionalism Inwnrd-Iooking Figure 1: Evaluation - a Cognitive Map. academy. Professional accountability is therefore funda- mental to self-evaluation and peer evaluation. Peer evalu- ation is indeed the oldest and most fundamental of all evaluation principles in higher education (Kogan, 1989, ρ 118). Finally, educational accountability also encompasses contractual accountability. Academic departments are therefore answerable to management in higher education as well as to political authorities. 3.2 Purpose and objectives of departmental evaluation Evidently the first step in any evaluation procedure is to determine the purpose and objectives of the evalu- ation process (Worthen & Slanders, 1987, ρ 147). This is not only dictated by logic but is vitally important be- cause educational evaluation is always undertaken within a particular context. Important contextual parameters can be determined by clarifying beforehand issues such as why evaluation is being undertaken, who the main audiences are, to what use evaluation findings may be put, what method of en- quiry should be used and who should do the evaluation (House, 1986, ρ 16). An analysis indicates that the purposes and objectives of departmental evaluation can be subdivided into three major categories. • Improvement (organisational learning) A major objective of departmental evaluation is to generate useful information for the improvement of departmental activities: information to management of the department for decision making and to do plan- ning. Seldin (1988, ρ 24) points out that there is no better reason to evaluate than to improve perform- ance. Evaluation data assist the faltering, motivate the tired and encourage the indecisive. Supplemen- tary (but related) to this, is the evaluation objective of incorporating into the operation of the academic department an ongoing concern for self-study and self-improvement (Kells, 1989 ρ 100). • Accreditation In this category, the objective of evaluation is to as- sess the extent to which accreditation standards are being met. It also provides useful written material for the evaluation team and formal reports for the accreditation authorities. • Accountability As previously indicated, a major objective of evalua- tion is to provide proof that resources are used optimally (contractual and moral accountability). An additional objective is to ascertain the quality of higher education provided by the academic depart- ment (professional accountability). Finally, the ob- jective of the evaluation process should be to provide a clear statement of the relationships between the goal and objectives of the academic department and the mission statement of the institution (contractual accountability). The importance of the conceptual and contextual is- sues discussed in the above paragraphs gains additional perspective for departmental evaluation when viewed against the background of specific concerns relevant for academic training for professional programmes. 3.3 Specific concerns relevant to evaluating education programmes for the professions The central feature that distinguishes a professional academic programme from general university courses can Die Suid-Afrikaanse Tydskrif vir Kommunikasieafwykings, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) 28 S.R. Hugo & W.M.J. Hugo be found in the definition of a profession. According to Hughes (1963, in Goodlad, 1985, ρ 7) the central feature of all professions is the motto "credat emptor" (let the buyer trust). Thus is the professional relation distinguished from markets in which the rule is "caveat emptor" (let the buyer beware). To develop an individual that embodies this code of conduct implies the incorporation of very specific aca- demic aims, strategies and procedures in the education programme. The one unchanging feature of professional training (and perhaps the main factor that distinguishes profes- sional academic training from other university pro- grammes) is unceasing movement towards new levels of performance. In this energizing process some of the fol- lowing characteristics are observable: • concern with mission / function clarification • mastery of theoretical knowledge • capacity to solve problems • credentially • public acceptance • ethical practices (Nowlen, 1988). This feature is usually utilized in the evaluation done by professional training bodies. These bodies play an im- portant role in establishing minimum standards for edu- cation and training, in controlling professional ethics and even in structuring university programmes. This may, on the one hand, be negative because it can function in di- rect opposition to the autonomy of academic institutions. On the other hand the professional bodies can contribute to the objective determination of worth of a specific de- partment because of their role as independent external observer of actual performance. There are moreover specific considerations appropri- ate to the evaluation of an academic department concerned with programmes for Speech Therapy and Audiology. Firstly, academic departments for Speech Therapy and Audiology in the RSA have the general characteristics of all academic departments in a developing country. Their main characteristic is the fact that up to now the depart- ments have failed to establish sustainable capacity for monitoring and evaluation. In consequence there is a lack of information on educational outcomes and costs. In an era of constrained resources for education development this implies that the effects of investments are difficult to gauge and lessons of experience difficult to accumulate (Middleton, et al., 1989, ρ 1). Secondly, on a ground roots level, evaluation of depart- ments for Speech Therapy and Audiology must take cer- tain practical characteristics into consideration. These in- clude: Π a low student-lecturer ratio and in conjunction with this a low student enrolment figure attributed to the policy of individual training and practical limitations; high grades of students and a low failure rate be- cause of student selection practised by most univer- sities; Π the difficulty in evaluating practical training of stu- dents where qualitative rather than quantitative measures are employed; • the fact that practical training is dependent on the availability and cooperation of outside agencies like hospitals, schools and clinics; • the influence on programmes by systems independ- ent of the academic institutions, such as the National Health Policy, the Professional Board for Speech Therapy and Audiology and the South African Medi- cal en Dental Council. 4 A MODEL FOR THE DEPARTMENTAL EVALU- ATION PROCESS 4.1 Conceptual basis A classification by Worthen and Sanders (1987, ρ 60) of different approaches to evaluation identifies six catego- ries: Π objective oriented approaches Π management oriented approaches • consumer oriented approaches • expertise oriented approaches • adversary oriented approaches • naturalistic- and participant-oriented approaches In analysing each of the above categories it became immediately obvious that the management approach would be a valuable basis for evaluation of a professional academic department for Speech Therapy and Audiology. There are three compelling reasons for this preference. The management approach to education which regards education as essentially an instrumental activity designed to bring about the achievement of specifiable and uncontro- versial educational goals is becoming increasingly domi- nant (Mortimore & Stone, 1990, ρ 69). This is especially true of tertiary institutions in general and universities in particular. The choice of the management approach for departmental evaluation is therefore congruent with a gen- eral trend. The second reason is that any department of Speech Therapy and Audiology is today, with the general movement at universities towards accreditation, reduced funding and unavailability of qualified academic staff (Strydom et al., 1991), involved in a struggle for survival. A management approach towards evaluation will provide valuable information for decision making (and decision makers) in this endeavour. Lastly the management ap- proach, if correctly understood and used in context, in- cludes many of the characteristics and principles of the other six approaches towards evaluation in education. It is therefore a comprehensive basis for departmental evalu- ation. I The questions now are: what is the management ap- proach to evaluation and how.can it be utilized by a de- partment of Speech Therapy and Audiology? In their analysis, Worthen and Sanders (1987, ρ 77) state that the management approach to evaluation is meant to serve decision makers. They continue to empha- sise that, by highlighting different levels of decision mak- ers (levels of management), this approach clarifies who will use evaluation results, how they will use them and what aspects of the system they will be making decisions about. In the normal management control procedure in an organisation information about actual performance is channelled to operational management, middle manage- ment and top management in accordance with the nature of the information and the control process. This is concep- tually identical to the principal characteristics of educa- The South African Journal of Communication Disorders, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) Departmental Evaluation: Speech Therapy and Audiology 29 1. Internal departmental self-study 2. External review or site visit by experts 3. Review of reports • Audience - operational management • Audience - middle management • Audience - top management • Formative evaluation • Formative / summative evaluation • Summative evaluation • Aimed at improvement • Improvement and reporting • Review reports generated by self- study and peer group • Departmental management • Peer group of professionals • Faculty / institutional management / political authorities • Professionally | accountable • Professionally / Con- tractually accountable • Contractual accountable Figure 2: Management Approach to Departmental Evaluation. tional evaluation referred to in paragraph 3.1.and can be illustrated in Figure 2. Firstly, according to the improvement purpose of evalu- ation, information is provided to departmental manage- ment to continually effect improvement. This is clearly formative evaluation aimed at enabling departmental management to be professionally accountable. Secondly, in accordance with the accreditation purpose, information about the department and the actual performance of it's activities is forwarded to some higher authority (faculty management level or even (institutional management level). This evaluation is clearly of a summative nature although it may contain some formative elements. Evalu- ation information at this level enables departmental man- agement to be contractuallyj accountable. Thirdly, in or- der to answer to moral accountability, evaluation infor- mation is provided to the highest management levels in tertiary education, the political authority that controls funding and also to other stakeholders such as professional boards and statutory controlling bodies. The control hier- archy typical of ordinary management control procedures is therefore also embedded in the basic principles of edu- cational evaluation. The implications of the preceding arguments for evalu- ating a department of Speech Therapy and Audiology us- ing the management approach, are in essence that a three- tier evaluation hierarchy with the following characteris- tics should be constructed (adapted from Kells, 1989, ρ 99): Π an internal departmental self study prepared by the professionals concerned, using an evaluation model with a management approach; Π an external review or site visit by experts in Speech Therapy and Audiology (peer evaluation) using the evaluation data generated by the internal study and other suitable evaluation procedures. This can be used in conjunction with a report from the Profes- sional Board for Speech Therapy and Audiology, but should not be based on such an evaluation alone; • a review of the internal departmental report(s) and the reports of the external peer evaluation by a pro- fessional group in the institution (faculty manage- ment or institutional management) and by the con- trolling board of the profession. 4.2 Proposed model The evaluation model proposed in this section is a con- ceptual model based on the general principles of educa- tional evaluation and on guidelines for departmental evaluation discussed in previous paragraphs. The model is illustrated in Figure 3, and in broad terms follows six logical steps. 4.2.1 Preplanning for evaluation The first step for departmental management is to iden- tify clearly the need for evaluation why the department should be evaluated. This decision can only be finalised once the persons and organisations who are likely to be interested in the evaluation results have been identified. Information generated by evaluating the department of Speech Therapy and Audiology is aimed at a three-tier audience departmental management; an external review board of experts; and at top management, at institutional and political levels. In general the reasons for evaluating an academic de- partment can include any of the following (Worthen & Sanders, 1987, ρ 8): • improving student development and performance • enhancing educator qualifications and performance • improving the departmental organisational structure • upgrading textbooks and other curriculum materi- als and products Π determining whether to continue, modify, expand or terminate an existing programme • reviewing and improving curriculum design and proc- esses. Die Suid-Afrikaanse Tydskrif vir Kommunikasieafwykings, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) 30 S.R. Hugo & W.M.J. Hugo Pre-planning • Need- ' . • Audience • Objectives" * / ν <'«*<%; , ProcessLOesignf - Activity areas Scheduling P e r s o n n e l Ϊ Designing Performance indicators Admin • Financing • Workload • User satis- faction PROCESS Γ "-"-^.OUTCOMES Ί Management • Planning • Organising •Control . r if (fj »,„<•, >> !j i ; r ii -may f & Proficiency • Teaching • Research •Commu- nity service Efficiency • Manpower •Finances •Facilities Comparison Objective information gathering Assessment o f actual performance Reporting Figure 3: Evaluation Model. This step also involves demarcating exactly what should be evaluated (for example, administration, programmes, organisational structures or research outputs) and iden- tifying the purposes and objectives of the evaluation. Fi- nally, the pre-planning phase should also include a sur- vey of the resources available for evaluation. 4.2.2 Evaluation process design Drafting a plan of action for departmental evaluation is the main objective of this step. Based on the demarca- tion exercise in the first step and on the purpose and ob- jectives of the evaluation, major departmental activity areas to be evaluated must be identified. , ^ Figure 3 indicates that departmental evaluation for Speech Therapy and Audiology can be subdivided into two major but interrelated entities: process evaluation and outcomes evaluation (adapted from Worthen & Sanders, 1987, ρ 77 and House 1986, ρ 17). On the process side, administrative evaluation is aimed at how well the administrative support activities of the The South African Journal of Communication Disorders, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) Departmental Evaluation: Speech Therapy and Audiology 31 department function. In addition, the evaluation is also aimed at determining how well the department is being managed. Process evaluation provides an overview of the ability of the department to function dynamically and flex- ibly and of its capacity to survive. In the outcomes section there are two major subdivi- sions or components. Evaluation procedures aimed at es- tablishing how well the department is doing what it is supposed to do, i.e. success in attaining its objectives (pro- ficiency also referred to as effectiveness) is one component. The second is the efficiency component, and it refers to how well the department is utilising the resources at its disposal. Figure 3 also shows that the major objectives of the department that must be realised are in the activity ar- eas of education, research and community service. The most important resources at the disposal of the depart- ment are manpower, finances and facilities. 4.2.3 Planning and designing performance indi- cators for evaluation Establishing performance measures and the concomi- tant norms, involves, in effect, designing yardsticks for the practical evaluation of departmental performance. A performance measure represents an integrated group or category of related departmental characteristics or activi- ties that jointly provide a picture of a specific departmen- tal outcome or result. Evaluation norms are a series of measurement techniques whereby quantitative values and/or relationships can be determined for specific depart- mental activities, several of which jointly depict depart- mental performance in terms of a performance measure (Hugo & Van Rooyen, 1990, ρ 295). It is essential that a large number of measures and norms be determined for each major activity area identi- fied in step 2. Here follow some measures and norms that may serve as examples. It isjhowever necessary that every department should develop jits own specific measures and norms based on internal characteristics and in conjunc- tion with the aims of the department. 4.2.3.1 Measures and norms for process evaluation • Administration j Measures and norms for administrative activities must be based on a comprehensiye description of the adminis- trative task. Measures and norms evaluating both profi- ciency and efficiency are essential. The following are ex- amples (adapted from Wirgin & Braskamp, 1987, ρ 29). 1. Financial consideration: 3. User satisfaction: administrative services: Total cost ratio Annual administrative budget Cost of typing = Total departmental budget Annual cost of typing staff Number of pages of study material produced 2. Administrative workload: Spread Number of assignments processed a month Average number of assignments processed per month Lead times = Average waiting time (hours or days) for typing, photocopying any month Travel arrangements Average waiting time for same tasks previous month Number of errors in travel arrangements Total number of requests for travel arrangements per time period • Management Evaluation of management attempts to obtain a pic- ture of how thoroughly the management elements of plan- ning, organising, coordinating and controlling are imple- mented. At managerial level the management-elements are therefore used as performance measures. Bearing in mind that the managerial process is an interactive proc- ess between the different elements of management, it is obviously not possible (nor desirable) to isolate the con- tribution to performance outputs of any particular element of management in the evaluation process. The evaluation of management performance is therefore largely conducted on a subjective basis with the aid of an evaluation sheet or questionnaire. 4.2.3.2 Measures and norms for outcomes 4.2.3.2.1 Proficiency Proficiency measures and norms are aimed at deter- mining how successful a department of Speech Therapy and Audiology is in attaining its goals. The most impor- tant of these goals are in the three areas, teaching, re- search and community services as indicated in Figure 3. Here follow examples of measures and norms to deter- mine proficiency on the attainment of goals on these three areas: • Teaching 1. Staff teaching performance: Student feedback = External examiners = Average assessment of students for a particular course Average assessment of students - all courses in the department Number of appointments of Μ + D examiners for individual staff members 2. Staff quality: Qualification ratio •• Total number of appointments of M + D Number of staff members with doctorates Total number of academic staff in department 3. Student performance: Completion ratio = Students: Completion of course in four years Students Completion of course in more than four years Die Suid-Afrikaanse Tydskrif vir Kommunikasieafwykings, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) 32 S.R. Hugo & W.M.J. Hugo Average pass mark: students in 1st, 2nd, 3rd and 4th year in ^ _ Speech Ther/Aud Average pass ma Average pass mark: students in the faculty man-hours) and a outputs of the department of Speech Therapy and Audiology (Hugo & Van Rooyen, 1990, ρ 298). Most academic departments have only three major re- sources available: manpower, finances and facilities (see Figure 3). • Research 1. Output: Publications = N u m h e r of accredited papers this year Average number of accredited papers in last 3 years Productivity =_ Total number of research publications this year Man-years allocated to research this year Student activity J^umi>er of active doctoral students /year Number of registered doctoral students /year 2. Quality: International ratio = - Composition of research = Number of internationally published articles this year Total number of articles published by the department Number of research papers: basic /applied / dissertation / monographs /accredited/ proceedings / reports 3. Impact: Citations = Total number of papers / publications published Number of citations in research publications in past 3 years Number of citations in this field: Science Citations Index • Community service 1. Involvement: Number of staff on boards of Representation associations/institutes this year Membership fees paid by department 2. Continued education: Number of students registered Certificate programmes = year Number of students the previous year 4.2.3.2.2 Efficiency Evaluation of efficiency is aimed at determining how well the department is utilising the scarce resources at its disposal. The measures applied for efficiency essen- tially depict the ratio between the inputs (for example, • Manpower 1. Workload: Input ratio = Distribution =. Number of postgraduate students qualified / year Number of professors in department / year Number of assignments marked by individual lecturer Average number of assignments for all lecturers 2. Development: Number of conferences attended Conference attendance = fry all staff Total number of academic staff Study leave = N u m b e r of staff on sabbatical /year Total number of academic staff 3. Growth of academic staff: Growth rate = Growth rate of academic staff / 5 years Growth rate of students registered/ 5 years Number of staff who acquired Qualification ratio ^doctoral degrees in the past 2 years Average number of staff in the department ι • Finances | I 1. Budget objectives: I Total budget allocation for conference Conferences = attendance | Number of conferences attended by staff/year I I Total budgeted cost for Photocopying = photocopying / year Actual budgeted cost for photocopying / year 2. Availability of finances: Increase in student subsidy units Ratio of financing = in past 3 years Increase iryactual departmental budget in past 3 years Total budget allocated to Total budget = department/year Total academic staff complement /year The South African Journal of Communication Disorders, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) Departmental Evaluation: Speech Therapy and Audiology 33 • Facilities 1. Utilisation: Computertime Library = Actual number of hours utilised on research mainframe computer Budgeted number of hours on research mainframe available /year Total number of loan requests for department /year Total number of academic staff in department 2. Availability: Computerisation = Number of PC's available in department Total number of academic staff in department Area of office space available to Accommodation = department Number of students registered in 4.2.4 Information gathering and quantifying depar- mental outcomes (results) In this step basic data are collected and analysed us- ing the measures and norms identified. In step 3 it may be necessary to review some of the measures and norms during this exercise because the process of gathering in- formation sometimes reveals relevant aspects of perform- ance that were ignored in the planning phase, or alterna- tively, the data required for specific measures and norms may not be available. 4.2.5 Evaluating actual departmental performance This step involves the actual evaluation, judgment or assessment of the evaluation data generated in step 4 in order to draw a conclusion about the worth or value of departmental performance, j Four major approaches can be used in judging or as- sessing actual departmental! performance (adapted from Worthen & Sanders, 1987 ρ 302 and Hugo & Van Rooyen, 1990, ρ 302): j • comparison referenced assessment performance is judged against other academic departments • criterion referenced assessment performance is com- pared to absolute standards • objective-referenced assessment actual performance compared to planned performance or specified perform- ance objectives • historically referenced assessment actual performance is compared to own previous performance or perform- ance trends based on historical data. 4.6 Reporting The evaluation report aims at providing evaluation in- formation (based on quantitative and qualitative elements of the evaluation procedure) to relevant authorities. As in the case of an academic department, these authorities may be departmental management, faculty management or po- litical authorities. It is essential that the report should be drafted with reference to the purpose and objectives of the evaluation plan. Based on evaluation information, it should also indicate areas or activities where improve- ment is recommended and should reflect possible future developments. 5 CONCLUSION The evaluation model proposed above is a conceptual model designed to be used for and by the a department of Speech Therapy and Audiology. It is essential to realise that in the implementation of the model, contextual in- fluences unique to each department will have to be con- sidered in structuring appropriate measures and norms. The principles on which the model is based are not new, and its only merit may perhaps be found in a logical struc- ture for the measurement of departmental performance. Measuring outcomes of educational practices is a mod- ern phenomenon. Valuing their worth is as old as phi- losophy itself. House, 1986 ρ 1 REFERENCES Departement van Nasionale Opvoeding. (1987). Akademiese standaarde by universiteit in die RSA. NASOP 02-129 (87/ 10). Pretoria. Emmet, T.A. & Bennett, J.B. (1988). Five indices of program vitality. The Department Advisor. 1(5), 6-8. Feldman, A.S. (1981). The challenge of autonomy. ASHA, 941- 945. Goedegebuure, L.C.J., Maasen, P.A.M. & Westerheijden, D.F. (ed.). (1990). Peer review and performance indicators. Utrech: Lemma. Goodlad, S. (1985). Educating for the professions. Surrey: SRHE & NFER-NELSON. House, E.R. (ed.). (1986). New directions in educational evalua- tion. London: Falmer Press. Hugo, W.M.J. & Van Rooyen, D.C. (1990). Purchasing and materials management. 2nd.ed. Pretoria: J.L. van Schaik. Kells, H.R. (1989). Self-Study Processes. 3rd edition. New York: Macmillan. Kogan, M. (1986). Education accountability. London: Hutchingson. Kogan, M. (ed.). (1989). Evaluating higher education. London: Jessica Kingsley. Lee, L.J. & Sampson, J.F. (1990). Apractical approach to program evaluation. Evaluation and program planning. 13, 157-164. Middleton, J., Terry, J. & Bloch, D. (1989). Building Educational Evaluation Capacity in Developing Countries. Washington: World Bank. Mortimore, P. & Stone, C. (1990). Measuring educational quality. British Journal of Educational Studies, xxxix(l), 69-82. Nichols, J.O. (1989). Institutional effectiveness and outcomes assessment implementation on campus: a practitioners handbook. New York: Agathon press. Nisbet, J. (1988). in The Scottish council for research in education, (ed.). The evaluation of educational programmes: methods, uses and benefits. Amsterdam: Swets & Zeitlinger. Nowlen, P.M. (1988). A new approach to continuing education for business and the professions. New York: Macmillan Publishing Company. Seldin, P. (1988). Evaluating and developing administrative performance: A practical guide for academic leaders. London: Jossey-Bass. Wirgin, J.F. & Braskamp, L.A. (1987). Evaluating adminis- trative services and programs. San Francisco: Jossey-Bass. Wolf, R.M. (1990). Evaluation in education: foundations of competency assessment and programme review. 3rd.ed. New York: Preager. Worthen, B.R. & Sanders, J.R. (1987). Educational evaluation: alternative approaches and guidelines. New York: Longman. Die Suid-Afrikaanse Tydskrif vir Kommunikasieafwykings, Vol. 40, 1993 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2) 34 ZODIAC 901, Another Star In The Madsen Constellation ZODIAC 901 offers the best of both worlds 1. Fast automatic performance of tympanometry and acoustic reflex testing for routine screening. 2. Complete impedance testing capabilities, preprogrammable automatic test sequences, coupled with the refinement of sensitive manual testing. Children's Headset Clinical tests Screening tests ZODIAC 901 lets you get the most out of middle-ear analysis = PTY LTD THE HEARING HEALTH CARE SPECIALISTS SWF HOUSE SOVEREIGN STREET BEDFORDVIEW 2008 SOUTH AFRICA P.O. BOX 630 . BEDFORDVIEW 2008 SOUTH AFRICA TEL: (011)622-1743 FAX: (011) 622-1306 Reg. No. 77/01577/07 Amtronix - breaking the sound barrier The South African Journal of Communication Disorders, Vol. 40, 199 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 2)