jamieson Selecting computer based authoring packages for training Duncan Jamieson Creative Learning Systems Peter Hosie Instructional Design and Evaluation Australia Choosing a Computer Based Training system is a daunting task. With in excess of one hundred major packages on offer, errors are costly in terms of scarce capital, opportunities foregone, and human resources wasted. Selection criteria must start with, and be matched to, learner needs and the training requirements of the organisation, and not be technology driven. This article will outline a process for the selection of a CBT package in line with these needs. Defining CBT What is a CBT authoring system? A useful starting point is to get some working definitions of CBT: • An interactive learning experience between a learner and a computer in which the computer provides the majority of the stimulus, the learner must respond, and the computer analyses the responses and provides feedback to the learner. (Gery, 1987, p6) • CAI is the direct use of the computer for the facilitation and certification of learning - that is, using the computer to make learning easier and more likely to occur (facilitation), as well a using the computer to create, record proving that learning has occurred (certification). (Burke,1982, p16) • The extensive use of the computer in the development, delivery and administration of training. (Jamieson, 1983) Jamieson and Hosie 189 Where to start? There are two basic ways of selecting a CBT authoring system. Perhaps the simplest way to approach the task is to get the brochures, evaluate the available systems, decide what is affordable and go ahead and purchase the goods. The other method is to identify what your organisation's training requirements are, and then begin. CBT authoring systems can not be all things to all people. Conversely, unless the tasks requiring training are compelling enough to warrant the selection of a dedicated system with specialist capabilities, a selected CBT system may be used in a number of training circumstances. Undoubtedly tomorrow's authoring systems will offer exciting possibilities. As enchanting as so-called artificial intelligence languages and such are, or will be, any evaluation is only concerned with what can be purchased and implemented now, not in a decade. Learner Requirements Learner Preferences If organisational training needs are the primary selecting criterion for a CBT authoring system then learner requirements must be pre-eminent. Every effort needs to be made to translate these requirements into courseware designs. To do this student preferences must first be established. In a recent review of CAI research, Dr Louis Wilson of the Hazeltine Corporation reported that students prefer: • to exercise control over the pace, sequence and strategy of instruction - i.e. freedom and flexibility; • colour to black and white. Judicious use of colour can result in substantial instructional gains; • feedback that is contingent upon the student's responses. Retention is improved when feedback is immediate for correct responses and assists students to locate and correct errors; • to progress at their own pace, to be given achievement summaries and have opportunities to review information to gain mastery; • to control the amount of learning by reviewing previous instructions, get extra assistance or attempt enrichment exercises; • to participate with parts of lessons by responding before progressing; • to have graphics inserted into text in a variety of ways. Appropriate use of graphics improves the comprehension of text and is a superior way of conveying information for procedural purposes involving the spatial location of object; • a creative approach to learning that is interactive and breaks complex operations and relations to simpler ideas. 190 Australian Journal of Educational Technology, 1992, 8(2) For a CBT authoring system to assist designers produce courseware akin to learner preferences it must have certain technological capabilities. Quality courseware is most effectively developed by providing courseware designers with the best tools available. As Kearsley (1983, p 14) observes computers have certain advantages in instructional settings, such as: permitting "students to learn at their own pace, individual learning styles are considered, resulting in increased student satisfaction. Most importantly, there is more control over learning materials and learning process". The Key: Interactivity "Interactivity is CBTs raison d'etre" (Gery, 1987, p42). Computers are going to be an important factor in all human learning because they make learning truly interactive for large numbers of learners on a cost efficient basis. Bork observes that "... Evidence suggests (Hartley, 1981) that student control over learner strategy is the most efficient approach to CAL design. Encouraging individual routes through information will assist students to become more actively involved in the learning process" (Bork, 1984, pp 1- 4). Learners "should be given as much control as possible over programs, or at least opportunities for regaining control at some stage of the instructional sequence" (Hosie, 1987, pp 5-10). Understanding and knowledge involve active processing rather than passive reception. Magnitude of the task Proliferation of CBT Systems Since 1982, Data Training (Weingarten Publications, Boston MA) has been evaluating CBT authoring systems and courseware vendors. Beginning with 12 systems in 1982, 68 systems in 1986, the list ballooned to 93 in 1987. Early in 1992, CBT Directions (Weingarten Publications, Boston MA) surveyed US vendors of authoring systems, finding at least 52 authoring systems being actively marketed. The authors of the survey admit they "probably didn't get every authoring system into this directory, and we don't pretend that it's comprehensive" (CBT Directions, 1992, p15). Parallelling this high level of activity has been the number of courseware vendors; the nine courseware vendors surveyed in 1982 reached 114 in 1986 and 150 in 1987. "The 1993 CBT Buyer's Guide" list 124 vendors providing a wide range of CBT courseware and services. The offerings continue to expand which is either paradise or a prospective client's nightmare. Jamieson and Hosie 191 According to Data Training (1987, p 5), "It isn't just the number of systems that keeps growing either, it's their capabilities ... the variety of features and options is fast outstripping our ability to report on them succinctly". Witness the growth in multimedia and electronic support features being incorporated into the latest versions of authoring systems. If the experts can't keep pace what hope is there for neophytes to the game? Weingarten Publications' has been publishing an annual "Guide to Computer-Aided Training" since 1982. These are of inestimable value for initially sorting which system or systems to choose. But such a publication does, by its own admission, have shortcomings. There is no attempt to rate the quality of the features mentioned, only their presence or absence - essentially a binary analysis. Despite this limitation such information saves considerable brain and legwork - permitting unsuitable products to be eliminated at an initial screening. Steps in selection of a CBT system Why do you need a mechanism? Clearly the task of evaluating a large number of authoring packages in detail is impossible. Given the large number of possibilities it quickly becomes clear that a process of selection, based on the learner's and the organisation's requirements, is necessary. The process outlined in Figure 1 will filter out those systems that do not meet the organisation's needs, and provides a mechanism to rank order candidate CBT systems. The cornerstone of the process is the clear identification of learning needs, and the necessary system capabilities to meet these needs. This is achieved through the formulation of a model CBT system, against which candidate systems can be assessed. Identification of Necessary System Characteristics As previously stated, the organisational learning needs are paramount in determining appropriate CBT development and delivery facilities. Tune spent in examining and articulating the forces that will impact upon CBT operations, and hence the possible CBT system configuration, will be repaid many times over in the latter stages of evaluation. The training required, the authoring environment, and the organisational climate, all contribute to the formulation of a model of a CBT system best suited to the needs of the learner and the organisation. There are a number of considerations when it comes to selecting an authoring system. The point to be stressed is that consideration must be given to a range of influences, not just the technology, viz: 192 Australian Journal of Educational Technology, 1992, 8(2) Figure 1: Steps in the Selection of a CBT System Jamieson and Hosie 193 • Training needs • Training audience • CBT System features, function and requirement • Authoring • Author training and support • Costs • Vendor capabilities • Available courseware • Consultants Bearing in mind the probable instructional design strategies, probable author capabilities, and other critical aspects of the development environment, the selection of authoring features and functions is a daunting task. Choosing an authoring features always requires trade-offs. Gery (1987, pp 81-87) offers some sound advice when is comes to the selection of authoring features. There are three interrelated trade-off dimensions in the selection of authoring systems: Productivity vs Creativity Power vs Simplicity Structure vs Freedom In addition to identifying the features and functions, these considerations will identify those features which are essential and those which are desirable. The essential features should constitute a minimum set of features and functionality to meet organisational and the learner's needs. Any system not meeting these requirements will be excluded from further consideration. Initial Screening The first step in finding a suitable CBT system is to filter out those systems that do not meet a set of essential criteria. These criteria are the minimum acceptable level of functionality, operation and performance. No attempt is made to rate those functions identified as essential, only to establish whether a system has them or not. These features must be derived from the earlier considerations and that they cover a number of aspects of the authoring system, not only authoring features. Instructional and Technical Assessment After deciding what features and functionality deserve inclusion, an evaluation instrument is constructed. A conscious effort is made not to delineate between hardware and software components in an effort to 194 Australian Journal of Educational Technology, 1992, 8(2) avoid becoming technically driven in the assessment of criteria. Therefore, software concerns predominate and hardware only becomes an issue when software performance is affected. The categories in this model are: Authoring Presentation Management Vendor Support A detailed instrument is at Appendix 1. There may be other considerations that an organisation may wish included in an evaluation instrument. However, the impetus for the inclusion of any feature or function must come from the earlier considerations, and not an arbitrary decision. In determining the suitability of any feature or function, two five point rating scales are employed. The "desirability' scale measured perceived learner and organisation features, with the rating scale providing a method of evaluating the quality of particular features. The two scores were multiplied to give a raw weighted score. The scales are: Desirability 5 = Very important 4 = Highly desirable 3 = Desired 2 = Desired but not required 1 = Present but not required Rating 5 = 100% match required 4 = 75% match required 3 = 25% match required 2 = 25% match required 1 = 0% match required The earlier decisions on features and functionality for the organisation and the learner must be reflected in the selection of the appropriate point on the item's desirability scale. In this example, maximum possible scores are: Authoring 410 points (40% of total possible score) Presentation 220 points (22% of total possible score) Management 135 points (13% of total possible score) Vendor Support 155 points (15% of total possible score) Jamieson and Hosie 195 These scores reflect the relative importance of each section in the selection of a CBT system. It must be stressed however, that these weightings are by way of illustration only. The selection of a system is influenced by different factors for every organisation. Accordingly, these factors will drive the inclusion of system features as well as the desirability rating for each feature. Gery (1987, p 76) makes the point to: • Take off your rose-coloured glasses and put on your hearing aid when listening to sales rep or 'born again' users. • Talk to peers who have looked at systems. A section for a "qualitative response" has been added (100 points or 10% of total possible score), consisting of: Subjective analysis 50 possible points Colleagues evaluation 20 possible points Literature comments 30 possible points Naturally this could be criticised, quite reasonably, for introducing too subjective a criteria into the evaluation. This claim can be readily countered by the observation that many CBT systems, as selected, use purely subjective evaluation methods. After all, an affinity with what learners find engaging should be part of the repartee of a CBT designer. The opinions of colleagues using the software should be sought where possible. However, because these opinions are subjective as well as secondhand, judicious weighting must given to this category. Reviews of the software under consideration should also be examined. The quality and quantity of these reviews varies, but on balance, they remain a valuable source of information and therefore should be included in the rating considerations. From this process, candidate systems can be rank ordered, with the systems with the highest score best suiting the needs of the organisation. Costings After rank ordering the potential CBT systems, it is necessary to overlay economic reality on the shortlist of desirable systems. There is no purpose in selecting the "best system" for instructional reasons only to find cost constraints forcing compromises. The intention of the cost analysis is to determine the system with least cost for producing specifically defined outputs. The analysis will identify 196 Australian Journal of Educational Technology, 1992, 8(2) variations in input for a common level of output. Systems not attaining this common level of output will be normalised with the costs involved in raising the solution to the required output level included in the costing for that solution. The analysis should be for a defined period from a base year, with costs expressed in terms of their discounted, or base year, values. This allows for systems with minimal initial expenditure and increasing recurring expenditures to be compared to systems with high initial capital investment but lower operating costs. The economic life is the length of time for which the system can be expected to yield benefits. In the context of CBT systems it is unwise to predict a very long economic life. Systems and technology are changing so rapidly, that even the most up to date system can be expected to be overtaken by new developments in a short time. Five years is probably a reasonable time for mainframe and mini based systems, or systems with a high degree of specialised application, and three years being a realistic figure for PC based systems. It is important to recall that the evaluation is for the most appropriate training tool to meet the needs of the organisation (including trainers and developers) and the learner, not necessarily the best computer system. As such, training considerations should predominate. The analysis, apart from considering capital and recurring costs, should deal not only with the basic software purchase, but should include: • Software (eg productivity and automated design tools) • IT equipment (eg multi media interfaces) • Non-IT equipment (eg video, audio, specialist equipment) • Consumables • Labour-based services (eg project management, methodology development) • Training (eg author, user, IT staff) • Accommodation Results gained during this costing stage may cause a change in the rank ordering of the desirable systems. Trials Having selected one, or a number of systems that have been judged to best meet the needs of the organisation, these candidate systems should be scrutinised under operational conditions, through a trialing process. There are three major activities in conducting trials for a CBT system: development of a prototype, pilot tests and field tests. It may not be necessary, nor expedient, nor cost effective, to conduct all three: closeness Jamieson and Hosie 197 of fit between systems, organisational circumstances and timings will dictate the amount of time and resources to be committed to this activity. The purpose of a prototype is to determine if the design of the system is appropriate: before progressing. In this case. the purpose of the prototype is to validate the system characteristics and capabilities identified earlier. Thus, a prototype represents some small portion of the system, usually selected as exemplary of the full-scale effort. A pilot test would involve trying out the system with a small group of "students". Such "students" are typically colleagues or subject matter experts who am not directly involved in the development of the system. The purpose of a pilot is to detect any major problems in the hardware, software, courseware, courseware development and project management process. The final stage of a trial is field testing in which the system is tried out in the actual training setting for which it is being selected. It may be necessary to review some of the earlier decisions (performance, system feature desirability, costing), based on the information gained. This review may alter the rank ordering of the systems. The outcome of this filtering process will be the identification of the most suitable authoring system, based on the organisation's and learners' needs. Conclusion The cornerstone of this process for selecting a CBT system is that training needs must always drive the selection of a CBT system. A system's features may be attractive, but if they do not facilitate the attainment of the learners' and organisation's training needs, CBT, as a training strategy, will not be a success. The critical success factors in Selecting a CBT system are: • Know the organisation's and the learner's educational, technological, and development needs and priorities; • Understand the definitions of features and scope of functionalities and how they relate to the needs of the organisation and the learner; • Establish selection criteria and identify what is paramount for the organisation; • Be prepared to make conscious trade-offs; • Don't expect everything in one package. Not all features will be of equal importance in the selection of a CBT System. While suppliers strive to increase the complexity of their offerings and the variety of the features offered, few systems offer all features and functionality. This leaves evaluators with a dilemma. On what basis is the most appropriate CBT system selected? 198 Australian Journal of Educational Technology, 1992, 8(2) Choosing an authoring system always involves trade-offs, whether this is realised or not. However, the foundation for a successful selection process is ensuring that all decisions on selection are based on the learning needs of the organisation and the learner. In making these trade offs, features and functions may be omitted, or their desirability changed, but this must be a conscious, considered action, rather than by oversight. In an process of selecting a CBT system, comparisons between systems must be as objective as possible. This can be achieved by measuring candidate systems against a definability/performance model derived from organisation and learner needs. References Bork, A. (1984). Computers and the Future: Education. Computer Education, 8(1), 1-4. Burke, R. (1982). CAI Sourcebook. New Jersey: Prentice-Hall. CBT Directions (1992). Comparing Authoring Systems: Where Do You Start? CBT Directions, May/June, 15-29. CBT Directions (1992). 1993 CBT Buyer's Guide. Boston, MA: Weingarten Publications. Data training. (1987). The 1987 Guide to Computer Based Training. Boston MA: Weingarten Publications. Data Training (1991). The 1991 Guide to Computer Based Training. Boston, MA: Weingarten Publications. Gery, G. (1987). Making CBT Happen: Prescription for Successful implementation of Computer Based Training in Your Organisation. Boston, MA: Weingarten Publications. Hosie, P. (1987). Adopting Interactive Videodisc Technology for Education. Educational Technology, July, pp. 5-10. Jamieson, D. (1983). Selecting a CBT System. Proceedings of the information Technology Month Seminar. Perth, Australia. Kearsley, G. (1983). Computer Based Training: A Guide to Selection and Implementation. Reading, MA: Addison-Wesley Publishing Company. Jamieson and Hosie 199 Appendix 1: CBT System Selection Authoring Desirability Rating 1 2 3 4 5 Menus or prompts for creating lessons 1 2 3 4 5 1 2 3 4 5 Authoring templates to support a full range of learning styles 1 2 3 4 5 1 2 3 4 5 Ability to bypass menus, prompts and templates for command level authoring 1 2 3 4 5 1 2 3 4 5 Unlimited use of the screen by author 1 2 3 4 5 1 2 3 4 5 Flexible branching under author control 1 2 3 4 5 1 2 3 4 5 Logical command structure 1 2 3 4 5 1 2 3 4 5 Full screen text editor (internal or external) 1 2 3 4 5 1 2 3 4 5 Import external text and graphics 1 2 3 4 5 1 2 3 4 5 Interface with other software for authoring 1 2 3 4 5 1 2 3 4 5 Use of non keyboard devices for inputting 1 2 3 4 5 1 2 3 4 5 Editor for unresolved branches 1 2 3 4 5 1 2 3 4 5 Easy screen partitioning and windowing 1 2 3 4 5 1 2 3 4 5 Spelling Checker 1 2 3 4 5 1 2 3 4 5 Multiple fonts and character sizes 1 2 3 4 5 1 2 3 4 5 Range of colour palettes 1 2 3 4 5 1 2 3 4 5 High resolution graphics 1 2 3 4 5 1 2 3 4 5 Ease of controlling video disc 1 2 3 4 5 1 2 3 4 5 Ease of controlling video tape 1 2 3 4 5 1 2 3 4 5 Ease of frame (video) and screen (text) integration 1 2 3 4 5 1 2 3 4 5 Ability for author to test lesson without leaving author system 1 2 3 4 5 1 2 3 4 5 Capable of creating lessons up to a maximum size of 360 kbytes 1 2 3 4 5 1 2 3 4 5 Ability to link several modules 1 2 3 4 5 Other Comments 200 Australian Journal of Educational Technology, 1992, 8(2) Presentation Desirability Rating 1 2 3 4 5 Flexibility in handling input from student 1 2 3 4 5 1 2 3 4 5 Range of response types 1 2 3 4 5 1 2 3 4 5 Use of non keyboard devices for input, selection and progression 1 2 3 4 5 1 2 3 4 5 Ability to use external devices other than video tape and disc 1 2 3 4 5 1 2 3 4 5 Ability to communicate with other software at runtime 1 2 3 4 5 1 2 3 4 5 Student resume facility 1 2 3 4 5 1 2 3 4 5 Easy back paging and progression 1 2 3 4 5 1 2 3 4 5 Student controlled branching 1 2 3 4 5 1 2 3 4 5 Hard copy of screen displays 1 2 3 4 5 1 2 3 4 5 Author/student communications facility 1 2 3 4 5 1 2 3 4 5 Help facility 1 2 3 4 5 1 2 3 4 5 Random test generation 1 2 3 4 5 1 2 3 4 5 Lesson progression and completion display 1 2 3 4 5 Other Comments Jamieson and Hosie 201 Management Desirability Rating 1 2 3 4 5 Student registration facility 1 2 3 4 5 1 2 3 4 5 Author controlled student tracking 1 2 3 4 5 1 2 3 4 5 Ability to record individual responses 1 2 3 4 5 1 2 3 4 5 Ability to record the number of help requests 1 2 3 4 5 1 2 3 4 5 Security facility that restricts access to performance data and courses 1 2 3 4 5 1 2 3 4 5 Ability to analyse individual student performance data 1 2 3 4 5 1 2 3 4 5 Ability to compare student performance 1 2 3 4 5 1 2 3 4 5 Ability to export student data for analysis 1 2 3 4 5 Other Comments Vendor Support Desirability Rating 1 2 3 4 5 Concise and easy to use documentation 1 2 3 4 5 1 2 3 4 5 On site author training 1 2 3 4 5 1 2 3 4 5 CBT based author training 1 2 3 4 5 1 2 3 4 5 Locally based technical support 1 2 3 4 5 1 2 3 4 5 Telephone support 1 2 3 4 5 1 2 3 4 5 Availability of local user groups 1 2 3 4 5 1 2 3 4 5 Implemented for a range of computer video configurations 1 2 3 4 5 1 2 3 4 5 Planned development cycle for system 1 2 3 4 5 1 2 3 4 5 License agreement non restrictive 1 2 3 4 5 Other Comments 202 Australian Journal of Educational Technology, 1992, 8(2) Qualitative Analysis Subjective Evaluation (50 points) Colleagues' Evaluation (20 points) Literature Review (30 points) Authors: Duncan Jamieson is a Training Consultant with Applied MultiMedia, currently based in Canberra. Duncan develops CBT and paper based self study materials, as well as advising on the identification, formulation, design and evaluation of Training in the fields of information Technology and Computing. Duncan has worked for the WA Department of Computing and Information Technology as a Human Resource Development Consultant, and has been the National Training Manager for the Australian Federal Department of Social Security and a CBT adviser. He is a member of the Australian institute of Training and Development (AITD), the Australian Computer Society (ACS), the Australian Society for Educational Technology (ASET) and the National Society for Performance and instruction (NSPI). Peter Hosie is currently a Senior Educational Development Officer with the Ngee Ann Polytechnic in Singapore. Peter has been the Executive Officer for the Telecommunications for Education and Training Taskforce, formed by the Western Australian Government to establish an enduring mechanism for the delivery of education and training using telecommunications. As an Education Officer with the WA Ministry of Education, Peter was involved in the development of an interactive videodisc package for career counselling. Please cite as: Jamieson, D. and Hosie, P. (1992). Selecting computer based authoring packages for training. Australian Journal of Educational Technology, 8(2), 188-202. http://www.ascilite.org.au/ajet/ajet8/jamieson.html