pirera Training needs assessment (and application) in an accounting firm Fiona Pirera and Daljit Singh KPMG Peat Marwick This paper outlines the implementation of a major project on assessing training needs in an Accounting firm (Touche Ross & Co). It discusses the various stages of the project including methodology selection, data collection, data analysis, reporting of results, application of results to course design and project evaluation. This project was carried out over a two year period (1987-1989). The quality of the training output is only as good as the training data input. If the training needs have not been properly identified, then both the training course and the training function are in jeopardy. (Rummier, 1987, p. 223) Background In 1987, Touche Ross & Co. was one of the eight largest accounting firms in Australia (a member of the group commonly referred to as the "Big Eight"). It had 1,100 partners and staff and operated from twenty offices in Australia. The largest division at Touche Ross was the Corporate Services Division, which primarily provided auditing services. Auditing refers to the various activities carried out by specialised accountants (ie. auditors), who gather and evaluate evidence for the purpose of forming an opinion on the financial statements of their clients. Touche Ross & Co, as a member firm of Touche Ross International, adopted a common international audit methodology known as the Touche Ross Audit Process (TRAP). TRAP represented a systematic approach to Pirera and Singh 137 auditing, with a sequence of three major phases which are noted in Figure 1 below. Each of these phases, in turn, consisted of several further stages of audit activity. Design of Audit Approach Assessment of Likely Errors in Financial Statements. Includes Tests of Controls (Systems work) Evaluation of Audit Evidence to Form Opinion on Year End Financial Statements (Balance Date Work) Figure 1: An overview of the Touche Ross Audit Process (TRAP) In late 1987, the partners in charge of the Corporate Services Division in each state (a group known as the Directors of Corporate Services), placed training high on the agenda of one of their regular six monthly meetings. The authors who were also present were then new appointees to their training positions at Touche Ross. Also present was the Chairman of the National Professional Development (NPD) Committee, the partner who had overall responsibility for training in the firm. The following views were expressed by the Directors: • Gaps appeared to exist in the overall audit training program • All course content needed to be reviewed and challenged against the knowledge and skills required by staff at each level. (They explicitly stated a need for a "skills matrix"). • The quality of training material and the effectiveness of the training process itself should also be reviewed. These comments made by the Directors were general in nature. The meeting established that management supported a project for a thorough re-examination of audit training and underlined the importance for the technical arm of the Corporate Services Department (NAA) and the training arm of the firm (NPD) to work closely together in carrying out the project. There was an expectation by the Directors that by the end of 1988, significant progress should be made towards meeting the above goals. 138 Australian Journal of Educational Technology, 1990, 6(2) Importantly, they were also willing to contribute the necessary resources for the project. The key question for the authors was where to proceed from there. We decided to review the literature on training needs assessment in order to base our work on a methodology which would provide us with relevant information to make the necessary decisions, but, naturally, at a reasonable level of cost. Literature review We commenced a literature search on the training needs assessment process, with particular emphasis on any reported research by Accounting firms on needs assessment. Several useful training texts and articles were located through this search. However, the hopes of finding specific articles on needs assessment for technical training (especially in the accounting field) were frustrated. None of the articles specifically addressed needs assessment in the context of an accounting firm. However, some of the frameworks noted were useful to our project and are described below. Newstrom and Lilyquist (1979) developed a contingency model to evaluate various needs assessment methods. They evaluated twelve methods on the basis of five selected criteria: • Employee involvement; • Management involvement; • Time required; • Costs; and • Relevant quantifiable data. The methods were ranked high, moderate or low on each of these five criteria. They noted that an ideal method would involve a high degree of employee and management involvement, require a modest investment of time and cost and would also produce a high degree of relevant quantifiable data useful for making decisions on training. Not surprisingly, none of the methods they evaluated produced the ideal rating on all five categories. Also, none of the methods had consistent poor ratings across all the five criteria. According to their evaluation, the strongest methods were performance appraisals and survey questionnaires. Newstrom and Lilyquist recommended that weaknesses in one method could be balanced by including other complementary methods and that trainers needed to weigh the criteria in terms of their importance to the organisation. Caffarella (1988) used the Newstrom and Lilyquist model to evaluate eight selected methods (see Figure 2). She excluded some methods such as Pirera and Singh 139 external assessment centres and included other methods which had not been considered by Newstrom and Lilyquist, such as critical incidents. Technique Criteria Emp Invlv Mgt Invlv Time Cost Quant data Observation L M M L L-M Survey M M H H H Interview H L H M M Group Meeting H M M L L Job Analysis L H H M H Tests M L M M H Critical Incident H H M L L Written Materials L L M L L Figure 2: Evaluation of eight selected techniques using Newstrom and Lilyquist Contingency Model (Caffarella, 1988) She described the eight methods she had chosen as those most widely in use, selected from major sources on data collection methods (Knowles, 1980; Steadman, 1980; Tracey, 1984; and Zemke and Kramlinger, 1982). According to Caffarella, the methods or as she called them the "techniques", could be used individually or in combination, depending on the objectives, people involved and the funding available for the needs assessment project. She also noted that no one method was suitable for all occasions. Caffarella also went further than Newstrom and Lilyquist by including the nature of the job as an additional factor that should also be considered in choosing needs assessments techniques. Her evaluation of the eight techniques, based on particular job classifications is noted in Figure 3 below. Technique Job classification Manual Technical Clerical Supervisory Managerial Surveys X X X Observations X X Interviews X X X X Group Meetings X X Job Analysis X X X X Tests X X Critical Incident Survey X X X X Written Materials X X X X X = appropriate technique for job classification Figure 3: Needs assessment techniques as related to job classifications (Caffarella, 1988) 140 Australian Journal of Educational Technology, 1990, 6(2) For assessing technical training needs, the recommended methods were surveys, interviews, job analysis (she subsumed task analysis under job analysis), tests, critical incident surveys and written materials (job performance documentation, performance appraisal records, policy and procedures manuals etc.). She added the caveat that the evaluation of these methods for particular job classifications should not be considered to be iron-clad as this assessment generally reflected needs assessment experience. Rummler (1987) provided a framework classifying four basic approaches to determining training needs, i.e. • training needs survey • competency study • task analysis • performance analysis Figure 4: Four approaches to determining training needs (Rummier, 1987) Pirera and Singh 141 Figure 4 details Rummler's framework with its four approaches and shows the extent to which these approaches link training input to performance output, which affects the quality of the information on training needs. Each of these approaches focuses on certain question(s), which highlight this link (or lack there of) between training input and performance output. Rummler acknowledged that the quality of the information needed to be balanced against the constraints of time, resources and client sophistication. He discussed each of the four approaches in relation to the sources of data collection, data analysis as well as the relative advantages and disadvantages. He also associated each of the four approaches with certain "primary" methods of data collection as noted in Figure 5. Approaches Primary Methods Survey Questionnaire Interview Group Discussions Critical Incident Observation Training Needs Survey X X X X Competency Study X X Task Analysis X X Performance Analysis X X Figure 5: Approaches to training needs assessment related to data collection methods (Rummier, 1987) We found this distinction between approaches and methods to be particularly helpful. The literature would often discuss an "approach" (eg. task analysis) alongside "a method" (eg. interviews) without making such a distinction. For instance, Figure 5 points out that interviews could be used as a primary method for three of the approaches. We also referred to the work by Kemp (1985) and Gropper and Ross (1987) on instructional design. Kemp (p. 15) in his reflections on instructional design made the following observation: . . . the instructional design process has not reached a level of scientific exactness. Many paths can be conceived to reach the same goals and objectives. Similarly, we noted Gropper and Ross's point that there was no single universally accepted instructional design model and that there were behavioural models, cognitive models and hybrids. However, Gropper and Ross did list the major tasks which they saw as part of an instructional design process used by many instructional designers and practitioners (see Table 1). According to Gropper and Ross, while instructional designers and practitioners may not always perform these tasks in exactly the same order, they would at their most systematic, perform all of these instructional design tasks. 142 Australian Journal of Educational Technology, 1990, 6(2) Table 1: Instructional design tasks (Gropper and Ross, 1987) 1. Needs Analysis 2. Task Description 3. Task Analysis 4. Sequencing 5. Stating Objectives 6. Developing Tests 7. Formulating Instructional Strategies 8. Developing Training Materials 9. Formative Evaluation Selection of methodology After carefully considering the various approaches to training needs assessment, we chose task analysis, a decision we believed was supported by the literature. Rummler had recommended this method as being particularly useful for jobs where a large number of people performed similar tasks, followed similar processes as well as performed very specific tasks (which fits in well with tasks typically required in auditing). Further, Caffarella had also included job analysis (in which she had included task analysis) as a recommended method for assessing technical training needs. A training needs survey was rejected as that would have been too imprecise and subjective. Similarly, competency study was also rejected as it would have been difficult to link this to job output. This was considered to be more suited for managerial and professional jobs with broad, difficult-to-define job responsibilities rather than jobs with specific well defined outputs (Rummier, p 238). We also considered whether performance analysis would be appropriate. We decided that performance analysis factors such as feedback, consequences, resources, attitudes etc. were being considered by the firm's: • Performance Appraisal System (feedback was provided after every audit assignment) • Counselling System (sessions were held every six months to discuss performance as well as career goals) We also considered the advantages and disadvantages of using task analysis. (Rummier, p.241). Firstly, the advantages were: 1. There would be a clear identification of required tasks and the associated knowledge and skills. 2. Tasks could be validated - ie. high "performers" do the tasks in this way. Pirera and Singh 143 3. Tasks are forms of output which could be measured (ie. training input could therefore be linked to task output). This was particularly important as training needed to be relevant to the auditing tasks actually being performed on the job. 4. Training recommendations would be based on fact and would not be affected by subjective factors. We saw this as being very important because of the significant resources that were likely to be committed to upgrade audit training. We also considered the disadvantages of using this approach. 1. It would be time consuming (60 - 90 days) and would require skill. The management of the Corporate Services Division and the National Professional Development Committee were highly supportive of implementing a thorough project, so time constraints were not a limiting factor. Regarding, "skill", Rummler recommended that one should have some training in task analysis before trying the approach. While the authors had not received any "formal" training as such, one of the authors did have prior experience in task analysis while working in the United Kingdom. We relied on the knowledge we had acquired from the literature search and advice from Touche Ross Management Consultants who had needs assessment experience. 2. The project would be "visible", which could have, depending on its outcome, positive or negative consequences for the training function. 3. While it would be possible to set priorities by surveying opinions on what were the critical tasks, the link may not necessarily be made between tasks and total performance. We noted this down as a factor to consider for data collection. 4. Task analysis would not address other factors in the performance environment which would affect how well a task was ultimately performed (eg. performance factors such as consequences, feedback, resources). As noted earlier, performance factors were being considered by the firm's performance appraisal and counselling system. However, this did not preclude us from considering the possibility of establishing closer links between the training and performance appraisal systems (which is discussed later in this paper). Data collection There were two main stages in the data collection process. Firstly, this involved the preparation of an audit training matrix, that is a comprehensive task listing, identifying the staff level when each task was first performed, the frequency of task performance and the importance of 144 Australian Journal of Educational Technology, 1990, 6(2) the task. Secondly, the data in the matrix had to be validated. Figure 6 provides an adapted extract of the audit training matrix. As Figure 6 has been adapted it only provides three of the six staff levels assessed for the tasks of auditing stock. Further, it does not include the clerical aspects of the tasks, which were also included in the matrix. AUDIT OF STOCK TASKS FIT Staff Levels Proposed Training 0-6 6-12 12-18 18-24 JUDGEMENTAL: Review Stocktake Instructions 31F X Month 5 Observe Count Procedures 21F X Month 5 Complete Questionnaire 21F X Month 5 Review Net Realisable Value (NRV) 21F X Month 17 Review Treatment of Variance 31F X Month 17 Apply Job Costing 41F X Month 17 Apply Std Costing 31F X Month 17 Figure 6: Extract of the Audit Training Matrix In order to prepare an initial task listing, audit procedures manuals were analysed to extract audit tasks and sub-tasks. A similar listing of accounting tasks from the UK was also used as a guide. This was supplemented by the authors audit experience, since there were a number of tasks which were not specifically addressed by the audit manuals. These were generally more specialised tasks such as auditing share options and hedged transactions. Having compiled the task listing, staff levels at which tasks were first performed were identified. The 'bands' could not be too broad eg. a cross in the 0- 12 months column could mean that a task could be performed in the first month of employment or towards the end of the first year of employment. Eventually, 6 month bands were used for the first two years of employment and 12 month bands were used for later years. As will be outlined later, this did require further refinement when training courses were linked to the other data collected. Three assessments were made in relation to each task (see columns marks FIT in fig. 6) • Frequency of task performance (F). ! Tasks were ranked on a scale of 1 to 5, with 1 denoting tasks performed more than once per audit while tasks ranked 5 were rarely performed on audits. • Importance of tasks (I) ! Tasks were also rated on the basis of their importance to job performance. Tasks were ranked on a scale of 1-3 with 1 denoting the Pirera and Singh 145 most important tasks while 3 denoted the least important task. In hindsight, a scale of 1-5 (consistent with the scale used for frequency of task performance) would have been more appropriate as using different scales had the potential to (and did indeed) cause confusion during the validation process. • Method of training (T) ! Tasks were also assessed in relation to the most appropriate method of training. 'F' denoted formal training, 'S' self-study and 'O' on-the-job training. By making these three assessments, we hoped to identify tasks critical to job performance and establish priorities for training. Important, frequently-performed tasks were more likely to feature in a formal training program while important but rarely-performed tasks might be covered in self-study modules or in specialist training programs. The draft 'Audit Training Matrix' document was over 40 pages long. It then required validation. We decided that the validation process should take place at the five capital city offices of the firm. This process would consist of the following steps: 1. Review of the draft matrix by two individuals at each level from audit assistant to audit partner. (There were six levels of staff identified for review purposes). Reviewers were to be "high" performers, so that the earliest level at which a task is first performed would be identified. 2. Once the results of the review process in (1) had been received and analysed, interviews would be held with a sample of the reviewers at selected offices. The matrix concept was explained to the firm's regional training managers at a meeting in March 1988. The sixty copies of the draft matrix were then distributed by these managers to the reviewers, who were selected by the training partners of the participating offices. The reviewers of the matrix were asked to identify: • whether the list of tasks was complete • whether the staff level at which each task was shown to be performed was correct; and • whether the frequency, importance and method of training for tasks had been correctly assessed. 146 Australian Journal of Educational Technology, 1990, 6(2) Reviewers were given a month to complete their review and requested to return the matrix, with their comments, to the regional training managers. By the end of April, 1988, forty copies of the matrices had been returned by the offices. This represented a response rate of sixty seven per cent. One of the offices decided to have a group meeting of all the reviewers from the various staff levels as the office preferred making a group response. Therefore, only one copy of the matrix was received from this office (representing group consensus) instead of the twelve anticipated. If we add another eleven responses to the forty received to take account of the group meeting, the effective response rate would be eighty five per cent. In summary, the data collection methods used were varied, and included survey questionnaire and interview. The matrix itself had been constructed on the basis of a review of written materials (firms technical manuals etc.) and the audit experience of the authors. Data analysis The responses received had various comments, ticks and crosses on the matrix document. Some reviews were more detailed than others. There were a few reviewers who had commented in detail on the first few pages, but then ran out of time/enthusiasm on later pages! All the amendments were plotted on a master copy of the matrix by using different coloured pens for each office. While the feedback did necessitate some extra tasks being added to the matrix, the majority of the comments and proposed amendments related to the staff level at which tasks were first performed. The original assessment of the timing of task performance on the draft matrix was often later than that identified by the reviewers. We decided to have follow up interviews with reviewers at the Adelaide, Melbourne and Sydney offices. Our main objective was to discuss the differences in the assessments of timing of task performance. We also wanted to interview the Directors of Corporate Services at those offices, to ensure that their views on training needs were given proper consideration and learn more about performance problems first hand. A number of specific questions were also planned on topics such as EDP training, interpersonal and communication skills and practical accounting skills. The interviewees were also to be encouraged to talk about their own perceptions of training in 'general'. The interview included a mixture of closed and open questions. As a result of the combined feedback from the questionnaire survey and the follow-up interviews, we were able to revise the matrix in a number of ways: Pirera and Singh 147 • A number of tasks were moved to an earlier level of staff performance. While some of these tasks were often being performed at a senior level on the larger audit jobs, they were also being performed at more junior levels on the smaller audit jobs (but at a lower degree of complexity). We decided that such tasks should be considered for inclusion in formal training at an earlier stage and should not be left to on-the-job training alone. Training for such tasks, at the more junior levels could cover the less complicated aspects of these tasks. We were aware of the danger of having training too early for these tasks, as knowledge and skills would be lost if not put into practice soon after the training. • We recognised the need to improve EDP and communication skills training. The final stage of the data analysis was to compare our current training program with the recommended training program. The matrix identified the training courses which currently addressed training for various tasks. We were pleased to note that most of the less specialised tasks were being covered by existing training courses. Current training courses did not adequately cover the more specialised auditing tasks and, as pointed out earlier, for some specialised and non-specialised tasks, the timing of actual performance of the tasks was generally earlier than anticipated. Some of the earlier timing also arose because audit staff were now performing some tasks which had once been done at higher staff levels. Therefore, the content of some courses was not matching the timing of performance of tasks on the job. We also reviewed the quality of the existing training material and found that there was room for improvement eg. there existed poorly developed visual aids and there was far too much emphasis on the lecture mode. Reporting the results Throughout the project, frequent reports of progress were being made to the Director of Corporate Services, Australia and the Chairman of the National Professional Development Committee. We decided that the final results of our task analysis should be reported to two separate groups: 1. Directors of Corporate Services !- a presentation at their meeting towards the end of 1988. 2. Regional Training Managers' meeting !- a presentation at their November 1988 meeting. 148 Australian Journal of Educational Technology, 1990, 6(2) Because of the massive amount of detail it contained, a revised version of the matrix would not have been a document that would have been easy to read and absorb. We compiled an executive summary describing the content of the courses which we proposed for the revised training program. We went through the audit matrix task listing and assigned each task to an appropriate course (where formal training was recommended). The two natural groupings of tasks were at the end of each year covering tasks of a systems/transaction audit type (Phase I and II work) and in May and June to cover tasks which were necessary to cope with the June end balance dates (i.e. Phase III work). (The majority of our clients had June year ends.). Our courses evolved as follows: 1. Graduate Induction Course (first week with firm) 2. Course in first month (M1) !- concentrating on systems (Phase II) work 3. Course in fifth month (M5) !- concentrating on balance date (Phase III) work 4. Course in twelfth month (M12) !- systems work 5. Course in 17th month (M17) !- balance date work 6. Course in 24th month (M24) !- systems work 7. Course in 29th month (M29) !- balance date work 8. Course in 36th month (M36) !- systems work 9. Course in 41st month (M41) !- balance date work The main topics which incorporated a number of tasks, were identified for each course - eg., audit of stock, engagement planning etc. The Directors of Corporate Services reacted with enthusiasm and support. Earlier in June 1988, a brief written report had been prepared for the Directors. Following their June 1988 meeting, they agreed to devote a substantial number of hours for the revision, update and improved design of training material based on the matrix. This time was to be from high calibre and experienced staff with good writing skills (to be nominated by the Directors). Also, these writers were to have an interest in training and be creative/imaginative. The Directors acknowledged that the quality of the rewrites would have a direct effect on the quality of the training. Pirera and Singh 149 Application to course design A decision was made to proceed systematically in course design by concentrating the design project on the first year training in 1989, followed by concentration on the second year training in 1990 etc. So it was to be a project that would involve instructional design work over a period of at least four years. Earlier in this paper, we had noted the major instructional design tasks which were part of the training development process outlined by Gropper and Ross. In terms of those tasks, the matrix effectively summarised the results of the needs analysis. It also provided a description of tasks, was developed using task analysis and provided a scheme for the sequencing (providing learning order) of related tasks. It also provided the basis for the other instructional design tasks such as specifying learning outcomes (objectives) and developing tests based on those objectives. In order to enable the nominated course writers to put the principles of course design into practice, we decided to conduct a one day training material design workshop. Two courses were held with a total of twenty course writers as participants. The workshop was to serve several purposes. Firstly, to ensure that we effectively communicated the basis of our proposed design model, including some principles on writing training material. We also wanted to explain the matrix in more detail and check that all the writers understood their respective assignments and would be able, in conjunction with us, to detail the knowledge and skill requirements for each task. Also, since sections of the various courses were to be written by different people from various states, we had to ensure that we adopted a uniform format for all training material. At the workshop there was emphasis on the importance of establishing well defined behavioural objectives which would form the basis of tests. There was also explanation of the difference between broad instructional aims, objectives and sub objectives. To illustrate the design of tests, a series of brief self-tests were included at the end of each of the workshop modules. Various types of questions were also discussed such as constructed response questions, recall type questions and recognition type questions etc. There was also discussion on the principles of developing case studies, emphasising the importance of simulating the practical world of the participants. Broad principles of formulating instructional strategy were also discussed. An excellent reference article on writing training material (by Silber and Stelnicki) was circulated for pre-reading prior to the 150 Australian Journal of Educational Technology, 1990, 6(2) workshop. This prompted much discussion and was one of the very few articles on writing training material which we were able to locate. An in-house expert on visual aids also gave a presentation on all the "do's" and "don'ts" of developing visual aids. While the workshop did provide the opportunity for all course writers to be fully briefed on their roles and covered some of the aspects of course design, the fact that it was only of one day's duration was undoubtedly a limiting factor. We were not expecting a one day design workshop to be, by itself, a guarantee of producing the best training material. However, we were trying to make the best use of our available resources and realised that, besides the workshop, we needed to institute a thorough quality review process of all the training material as it was being produced. Course writers were to write the material by specified deadlines on a part- time basis at their respective offices. The deadlines were set to allow time for review, amendment of course materials and pilot testing. Evaluation Unfortunately material received from the 'part-time' course writers was sometimes hurriedly put together and consequently the quality was not always as high as we had hoped. We had wanted "high performers" to write the material. However, these people were inevitably also busy people who had a large amount of client work, with tight deadlines. This made it difficult for our deadlines on reviews and pilot runs to be met. To overcome such problems in future, we decided that for the 1989 and 1990 rewrite of material, we would be recommending that individuals be seconded to the National function for specified periods of time so they could concentrate on writing material. Nevertheless, the material we managed to develop did, on an overall basis, "pass the test". Participants reacted very favourably to the new and revised courses based on the matrix. They enjoyed the increased level of participant involvement which was being built into our audit training and also, importantly, were finding the training to be both relevant and timely. We were able to obtain detailed participant reaction as a result of redesigning our course evaluation forms. This required participants to appraise each and every course module on pre-set criteria rather than evaluate the course as a 'whole' (as had previously been the case). The reaction of the trainers was also monitored through the use of newly designed instructor evaluation forms which sought detailed data on time allocations for particular modules, testing of participants to check learning, quality of training material etc. Management was pleased with the progress of the project as by the end of 1988 considerable progress had been made. We had in place a Pirera and Singh 151 comprehensive design document (the audit matrix) which covered all auditing tasks. Course content could be readily referenced to tasks done on the job and content concentrated on the prioritised tasks. Course design principles were being brought to the fore in our training methodology and course evaluation was being improved through our efforts to develop tests and improve the quality of the reaction data which was being collected (based on participant and instructor reaction). To address the issue of transfer of training to the job, another follow-up project was started in 1989. This involved the development of the Professional Development (PD) Passport which was an attempt to formalise the structure of work experience and promotions for all members of the Corporate Services Department. The plan centred around the idea that staff would have to achieve satisfactory results in performing specified tasks as part of their career development before progressing to the next level. Once the specified tasks had been logged onto the individual's passport, they would be considered for promotion. Passport "tasks" would be listed in an activities catalogue (drawn from the matrix) which would be a database accessible to all staff. Planning would be achieved as part of regular planning sessions. The staff member and the planning partner, or adviser would be able to access the activities catalogue to plan the type of work and specific tasks to be achieved by the staff member in the next few months. Similarly, recording the achievement of a task would be integrated with the existing staff performance appraisal system. Staff members would check the record of the activities which had been completed and have those verified by their appraiser. In our view, specifying job tasks and having the training for such tasks (audit training matrix) linked with the performance appraisal system (via the PD Passport) would have enabled a closer relationship to be established between the training and personnel systems. This longer term goal would have enabled us to be conceptually closer to Rummler's performance analysis approach and have provided stronger links between training input and performance output. Tasks would become the standards for performance and the performance context of the trainee would also be considered. Further work on implementing the matrix and developing the PD Passport project was suspended in late 1989. Touche Ross & Co (Australia) had by then entered into negotiations to merge with another major accounting firm. In 1990, Touche Ross & Co (Australia) merged with KPMG Peat Marwick. The methodology and results of this project will be considered for further application by the merged firm. 152 Australian Journal of Educational Technology, 1990, 6(2) References Caffarella, R. S. (1988). Program Development and Evaluation Resource Book for Trainers. New York: John Wiley and Sons. Gropper, G.L. and P. Ross (1987). Instructional Design. In R. L Craig (Ed). Training and Development Handbook. Third Edition, American Society for Training and Development, pp. 195-216. Kemp, J. E. (1985). The Instructional Design Process. New York: Harper and Row. Knowles, M. S. (1980). The Modern Practice of Adult Education. New York: Cambridge Newstrom, J. W. and J.M. Lilyquist (1979). Selecting Needs Analysis Methods. Training and Development Journal, October, pp. 52-56 Rummler, G.A. (1987). Determining Needs. In R. L. Craig (Ed). Training and Development Handbook. Third Edition, American Society for Training and Development, pp. 217-247. Silber, K. H. and M. B. Stelnicki (1987). Writing Training Materials, In R. L. Craig (Ed.). Training and Development Handbook. Third Edition, American Society for Training and Development, pp. 263-285. Steadman, S. V. (1980). Learning to Select a Needs Assessment Strategy. Training and Development Journal, January, pp. 56 64. Tracey, W. R. (1984). Designing Training and Development Systems. Rev. Ed. New York: AMACOM Zemke, R. and T. Kramlinger (1982). Figuring Things Out: A Trainer's Guide to Needs and Task Analysis. Reading, MA: Addison-Wesley. Authors: Fiona Pirera, Senior Manager (Audit Division) with KPMG Peat Marwick, Melbourne was formerly Manager, National Accounting and Audit for Touche Ross & Co, Australia. Daljit Singh, Senior Consultant (Professional Development) with the National Office of KPMG Peat Marwick, Sydney was formerly Director of National Professional Development for Touche Ross & Co, Australia. Please cite as: Pirera, F. and Singh, D. (1990). Training needs assessment (and application) in an accounting firm. Australian Journal of Educational Technology, 6(2), 136-152. http://www.ascilite.org.au/ajet/ajet6/pirera.html