pearson.pdf Australian Journal of Educational Technology 2003, 19(3), 275-292 Developing inclusive practices: Evaluation of a staff development course in accessibility Elaine Pearson University of Teesside Tony Koppi University of New South Wales After three iterations of a mixed mode (online and face to face) staff development course, an evaluation was carried out to determine the participants’ perceptions of its quality as an online environment, its effectiveness in introducing staff to issues of accessibility and its success in enabling staff to develop skills in accessible course design. This course was developed for a series of staff development cohorts each consisting of a mixture of academic staff, IT staff, and library staff. It aimed to develop expertise in the design of inclusive and accessible learning environments, to apply this knowledge in the development of the participants' own projects, and to encourage other staff to consider accessibility issues when designing online courses and resources. Methods used for evaluation included feedback during the course, responses immediately following the course, and a review of any lasting impact upon practice some time after the course. We also employed the services of a student who is blind to evaluate the course for accessibility following changes made to the original course. The evaluation instruments used were an online discussion forum, paper based evaluation, an online survey and an email request to the first two cohorts six to twelve months after the modules. The main lessons learned with respect to course design were concerned with communication of instructions, use of checklists as a learning guide, timing of the course during a larger staff development program, and availability of checking tools for practice purposes. Introduction A mixed mode course in accessible course design was developed as a component of an Innovative Teaching and Educational Technology (ITET) staff development program. The program is offered to Teaching Fellows at 276 Australian Journal of Educational Technology, 2003, 19(3) the University of New South Wales and consists of a six-month program during which time Fellows are bought out of their school or faculty commitments. A cohort comprises 15 Fellows who engage in four types of weekly activities: • Group topics: two or three Fellows develop and facilitate a four-hour workshop for the cohort on a topical learning and teaching issue (such as graduate attributes, and peer mentoring). • Project development: each Fellow has a project concerned with developing an exemplary online learning program for students. • Project groups: a diverse group of five Fellows form an action learning support group to assist each other in their project design and development. • Online activities: about three days per week are spent with the Fellows being online students and engaging in online activities concerned with educational theory and its application to creating online learning environments The Fellows are a diverse group of academic staff, IT staff and Library staff to ensure that the full range of people engaged with student learning is included. Three cohorts have now completed the ITET program; the first program was entirely face to face; the second one included some online discussions and resources; and the third was mixed mode with online activities and discussions, all being supported by face to face workshops. The accessible course design module was developed initially as a series of face to face workshops for the first ITET cohort and then adapted and transferred into an online course. The module is taught in mixed mode with activities and discussions online, laboratory supported activities with specialist software (this can be done on the participants own computer with the relevant software) and face to face workshops. The accessibility module is presented as a stand alone component separate to the main ITET Fellowship program. The reasons for this are two fold: firstly the online accessibility component was developed prior to the online course for the rest of the program, and secondly it is intended as a model of a complete program studied for credit. After three iterations of the module, it was evaluated in terms of the participants’ perceptions of its quality as an online environment, its effectiveness in introducing staff to issues of accessibility and its success in enabling staff to develop skills in accessible course design. Pearson and Koppi 277 Evaluation included a feedback discussion forum in the online course, a workshop feedback questionnaire, a survey in the online course, and an email request for lasting impressions from the first two cohorts. A student who is blind was employed to evaluate the course for accessibility, following changes made to the original course. The results of the evaluation enabled revision of the course for future staff development and additional recommendations for those considering including training in accessibility as part of their staff development activities. Rationale for the module and its evaluation The module was included in the Teaching Fellows program as part of an ongoing research and development partnership between the Special Needs Computing Research Unit (SNCRU) at the University of Teesside (UoT) in the UK, and the Educational Development and Technology Centre (EDTeC) at the University New South Wales (UNSW), Australia. It is part of a strategy to introduce and encourage inclusive practices, particularly in online learning, across the teaching and learning community. A number of initiatives have developed from the partnership, including “Guidelines for Accessible Online Courses” (Pearson and Koppi, 2001), workshops and seminars for other Australian universities, and a grant from the Higher Education Funding Council (HEFCE) to provide staff development programs on accessible course design for universities in the UK. For this reason the accessible course design module was evaluated as a discreet module in the Teaching Fellow program, in an attempt to determine its effectiveness and appropriateness not only for the Teaching Fellows program at UNSW, but also for transfer to and inclusion in other staff development programs at UNSW, UoT, and other Australian and UK universities. Module aims and outcomes The aim of the accessibility module is to enable staff to develop expertise in the design of inclusive and accessible learning environments, to apply their knowledge in the development of their own projects and encourage other staff to consider accessibility issues when designing online courses and resources. It is scheduled at the stage in the ITET program when the Fellows are considering their learning designs for their individual projects. The stated learning outcomes for the module are as follows: At the end of the module you will be able to: 1. Discuss the issues relevant to accessibility of online learning for people with disabilities. 278 Australian Journal of Educational Technology, 2003, 19(3) 2. Appraise the use and application of assistive technologies. 3. Analyse barriers to accessibility in existing web sites and online courses. 4. Formulate learner centred design strategies for accessible online courseware. 5. Demonstrate skills in the use of relevant guidelines and accessibility checking mechanisms. 6. Apply skills in the design and development of accessible and inclusive online courseware in your own projects. Face to face workshop prototypes Face to face workshops on accessibility were developed for the first cohort of Fellows (ITET1), with the intention of using the workshop tasks as prototypes for the online tasks. The workshops revealed how the learning tasks worked, how long they took, what the problems were, and obtained feedback from the participants. This was, in effect, a pre-evaluation of the intended online tasks. The online course was modelled along the lines of the workshop: orientation introduction, activities, reporting back, with support from the workshop facilitator (Koppi and Pearson, 2002). The results of the workshop experience enabled refinement of the activities for the online course, mainly by adding more support and tasks to provide a background and orientation to the issues of accessible course design. It also became apparent how much more valuable an online course can be than an ephemeral face to face workshop, which disappears without visible trace. The online course can be revisited and re-examined after further learning or application of learning has occurred. Access to essential information, resources and links to available software is easier, more efficient and available wherever the student has access to the computer. Fleeting ideas in a workshop, too soon gone because of the pace, can be explored in the online course in discussions and in student presentations that can remain on the site for all participants to access (Bunker and Vardi, 2001). The course is kept open to participants after the course has ended to enable them to re-visit and explore at their own pace, or when the material is needed in their own online course design. The advantages of face to face though cannot be denied – the time commitment is made and colleagues are there on hand for immediate discussion, however brief. It seems that commitment to an online course can be problematic for busy people because it is too easy not to set the time aside (Forsyth, 2001). Pearson and Koppi 279 Learning design The design of the course was based on the cognitive apprenticeship model (Brandt et al., 1993) as detailed in Koppi and Pearson (2002). The cognitive apprenticeship model (Collins, 1988) involves the use of modeling, coaching, reflecting on performance, articulation and application. Knowledge and skills are taught in contexts that reflect how the knowledge will be used in real life situations. Following this model, the Fellows first engage with studying how an expert approaches the problem (modelling), in the form of a video interview. The participants then engage with the issues identified by the expert by being placed into groups of three to undertake a range of activities (coaching). These activities include research; the opportunity to experience and develop skills in each of the five themes of the module: the law, guidelines, designing accessible documents, assistive technology and checking mechanisms (Pearson and Koppi 2002) and group discussion (reflection). Each small group is given responsibility for investigating one theme in depth to share with the other participants (articulation). The final part of the application of the cognitive apprenticeship model is when the participants evaluate an online learning environment by applying what they have learned from their activities and discussions (application). Although the participants were not being assessed, we wanted to ensure there was alignment of objectives, activities and outcomes, that we used active learning methods, that there were opportunities for evaluation and review, and that participants were encouraged to reflect on and apply their learning to their own individual situations (McAlpine et al., 2001). The online course is divided into three sections: orientation (including expert perspective video interview and videos of a student who is blind accessing online courses and documents, supported by resources and reflective questions); a series of learning activities; and a final whole group activity where the participants apply their new knowledge. The online components are supplemented by two face to face workshops. Evaluation methodology As the creation of web based learning environments continues to grow, so too does the need for their systematic evaluation (Owston, 2000). This course has been evaluated to determine the extent to which it meets the learning outcomes of developing the learners’ skills, awareness and understanding of accessible design; and its success with instructional design elements – structure, support, and the quality and relevance of the 280 Australian Journal of Educational Technology, 2003, 19(3) activities. Another concern was whether the accessibility component had any lasting impact on attitudes and practice (Lockee et al, 2002). A combination of evaluation instruments was used (Table 1) to try to be responsive to feedback during the course, to gather perceptions of the participants as they experienced it, to elicit responses immediately following the course, and to determine whether the participants' experiences had any lasting impact in practice (Patton, 2002). Following some changes to the original design, the module was re-evaluated in terms of its accessibility by Darren, a student who is blind. The instruments used were an online discussion forum through which the participants could give feedback on aspects of the course during the module; a short paper based evaluation at the end of the face to face workshops; an online survey that participants were encouraged to complete at the end of the module; and an email request to the first two cohorts six to twelve months after the modules to determine whether there had been any lasting effect on attitude and practice. Table 1: Methods used to evaluate the accessible course design module Evaluation instrument Timing of evaluation Evaluation type Feedback discussion forum During accessibility module Qualitative comments Paper based evaluation End of face to face workshops Likert scaling and free text questionnaire Online survey End of module Likert scaling and free text questionnaire Email survey Six to twelve months after end of Fellowship program Qualitative comments Accessibility review A course in accessible course design should be an exemplar of accessibility in itself. The course was checked for accessibility while it was being developed using a number of software tools and manual checking methods. These tools included the built in accessibility checker provided in Dreamweaver; the automated checking software Bobby (http://bobby.watchfire.com/bobby/html/en/index.jsp), that allows testing of web pages for accessibility in relation to existing guidelines, such as Section 508 and the W3C Web Content Accessibility Guidelines; A- Prompt (http://aprompt.snow.utoronto.ca/index.html), that provides an automated report and fixing wizard on the accessibility of a web page; and Wave (http://wave.webaim.org/index.jsp) which provides a graphically Pearson and Koppi 281 supported check of accessibility features. Manual checks included checking for navigational and structural clarity, and ‘reading’ the pages using the Jaws screen reading software. As with all usability testing, it is good practice to have a user with disability check the course if possible, because problems can be identified that wouldn’t be picked up by automated checks (Neilson 2000). For that reason Darren, a student who is blind and uses the Jaws screen reader to access the computer, and who is also an expert in accessibility, reviewed the course. The course was developed in WebCT 3.7 and there are some accessibility problems associated with the software that the course designer cannot change, such as the use of frames, java applets for discussion and chat forum, and the inconsistent naming of default links (Pearson and Koppi 2001, Alexander 2002). This in itself makes it important that the course is carefully designed to maximise accessibility. The course received a relatively clean bill of health for its accessibility, though there were some problems identified by Darren that need to be addressed. Darren found the section giving accessibility information very useful, but advised that it could be extended and improved. The section contains advice on how to turn off the navigation bar, a link to Adobe Acrobat Reader 5, general accessibility information about the site itself and hints on navigation. Darren suggested making the advice more generic (i.e. not specifically for those using screen readers) and including some information about the design constraints for accessibility of WebCT (e.g. inconsistent naming of default links, link name repeated twice where there is an icon and a text description). The second suggestion related to the ambiguous naming of some links. This occurred in the section introducing the videos. Here the user can choose to open the video at various download capacities depending on their connection (broadband, ISDN or modem). The links here are named, for instance “Broadband accessing the internet, ISDN accessing the internet” and so on. This could be confusing and it was suggested that this section should be restructured and renamed to make it clear what these links mean. Although these are relatively minor issues, it is nevertheless important to try to be responsive to any accessibility issues. Feedback discussion forum The feedback discussion forum was intended to enable participants to raise issues or make comments during the module so that where possible, the feedback could be acted on immediately and adjustments made, so it was as much about student support as feedback. Seventeen messages 282 Australian Journal of Educational Technology, 2003, 19(3) were posted to the feedback forum during the week of the ITET2 module and ten posted during the week of the ITET3 module. Feedback fell into three main types of comments: access problems, information overload and comments. Access problems, such as broken links and software problems were dealt with immediately to improve the student experience. The information overload comments indicated that some participants were having trouble working out exactly what they should be doing. There were two main causes of this problem. The module had been designed with a dual purpose, for Masters students studying the module for credit at University of Teesside and for the ITET fellows for staff development. Consequently, the references to assessment for the Masters students were confusing for some ITET Fellows. Secondly, in an attempt to provide a ‘belt and braces’ approach to giving support and information to participants, course information online was reinforced and supplemented with email messages. This was not always appreciated and instead of providing clarity, it resulted in confusion for some who stated that they would prefer all instructions and communication to be in one place. The third type of response was where participants indicated that they were enjoying the learning experience and that they wanted to organise their own hands on activities to present their findings to their colleagues. In response to the feedback from the second cohort (ITET2), it was decided for the third cohort (ITET 3) that apart from an initial introductory email, all communication and information would take place within the site. Any new information or reminders were posted using the ‘student tips’ tool in WebCT so that they would see it as they logged on to the course. To improve clarity and reduce confusion all reference to assessment in the activities was removed and the terminology used related directly to the staff development course. However, the assessment criteria that would be used for the Masters students was retained and renamed ‘Checklist for Learning’ to help participants to reflect on the effectiveness of the activities in developing their knowledge of accessibility. Workshop evaluations Two face to face workshops were held during the module, these workshops were supported by the online activities. In the first workshop participants fed back to the group the results of the activity assigned to them. Some used the time to introduce tasks related to the topic while others presented and talked about their work. They all submitted presentations based on their findings to the Student Presentation area of the online module. The second workshop was held after the final activity during which the whole group used the skills they had developed to carry out an accessibility analysis of a sample online course. During this workshop the participants shared their findings then reflected on how Pearson and Koppi 283 their experiences would affect their own projects, which are concerned with developing online courses. Participants were asked seven questions relating to their perceptions of the course that required a value response. The responses showed that the participants thought the workshops were ‘really good’ (six of ten respondents), that they learned a lot of new information and skills, and that they recommend the workshops be repeated for the next ITET cohort. The Fellows felt that the online course integrated well with the face to face workshops (seven of the ten) but that the amount of time allocated for the activities that contributed to the workshop was insufficient for the work involved. Although some felt that they would be able to use the things they had learned in their own projects (four of the ten) two were not sure at that stage how useful it would be and one felt it was not relevant. In response to a request for specific suggestions about the timing or structure of the activities, the responses indicated that about half would prefer the course to be fully integrated with the rest of the ITET course: Integrating it with the other ITET activities would make it more highly valued. Asked to comment specifically on positive or negative aspects of the workshops the general tenor was that the workshops were useful and should be more generally available to academic staff, the participants particularly appreciated the opportunity to use specialist software: Access to the specialist software is very helpful to contextualise the relationship between accessibility and online learning Asked for general comments or suggestions for improvement, the responses indicated that they would prefer to have more time to explore topics and that it should come earlier in the ITET program: It would be great to have these sessions earlier and over say two weeks so at least two topics/activities could be explored by each group. Other respondents clearly found the module beneficial, typical responses being: I really enjoyed this topic which I found full of usable ideas/ content. Well structured and I learned much from the activities. Online survey A number of online evaluation instruments have been developed for formative, summative and comparative purposes. Such instruments may 284 Australian Journal of Educational Technology, 2003, 19(3) provide evaluation criteria specific to evaluation of web sources (Beck, 1997), guidelines to assist in the choice of online course (Barker, 2001), or criteria to assess the quality of a course in terms of pedagogical design (Alley, 2000). We required an evaluation instrument that could be adapted for use by the participants in the course rather than for the course designers, that would compliment the other evaluation instruments being used (workshop evaluations and post-study survey) and provide information that was specific to the online environment, The framework on which the online evaluation was based is that developed by Herrington et al (2001) and encompasses the complete online setting: pedagogies, the learning activities which underpin the unit; resources, the content and information which are provided for the learners; delivery strategies, issues associated with the way in which the course is delivered to the learners. Although this evaluation instrument was designed for course developers, the checkpoints in the framework could usefully be adapted to an online survey to determine the participants’ perceptions of the success of the course. Eighteen questions were developed to evaluate the quality of the online course in terms of the pedagogical aspects, resources to support the course and the delivery strategies employed. The questions reflected the Herrington checklist examples. The participants were also invited to make additional comments at the end of the survey. Sixteen of the thirty fellows enrolled on ITET2 and ITET3 responded to the online survey. Pedagogies Six questions were formulated to evaluate the participants’ views of the learning activities in the online course relating to the authenticity of the tasks, opportunities for collaboration, focus on a learner centred environment and engagement. Questions relating to assessment were omitted. The questions were framed as statements and participants were asked to indicate the extent to which they agreed with the statements: • The activities enabled you to work collaboratively. • The activities supported your learning rather than being directed. • The activities challenged and motivated you. • The activities are realistic and sufficiently complex. • The topics are relevant to the subject matter. • There is a clear relationship between activities and learning outcomes. Pearson and Koppi 285 The responses indicated that on average 86% of the participants agreed or strongly agreed with the statements across the six questions. Typical comments were: I particularly enjoyed the tasks and what I learned. It was great to have the time to actually experience the material and begin to appreciate some of the access issues for specific groups in the student cohort. It appeared that the nature of the activities which required participants to collaboratively explore, experiment with and experience software, checking tools and assistive technologies, as well as investigate more theoretical aspects and apply them to their own situations provided a motivating, realistic and contextualised learning environment (Grabinger and Dunlap, 2002) One participant did, however, note that the workshop presentations were successful because they were face to face: The presentations worked very well. The only reason we could do the activity was because it was face to face – online collaborations take much longer. Resources Seven statements were framed to determine the participants’ views on resources in terms of their organisation, relevancy, currency, variety, accessibility and inclusiveness: • Resources are organised to make them easily accessible. • Resources are current and relevant. • A variety of media is used to support data sources. • Videos and other media used in the course are relevant to the purpose. • Activities and materials are accessible to people with disabilities. • Activities and materials avoid gender and culturally exclusive terms. • There is sufficient online support from the tutors. Again, participants indicated a high degree of satisfaction with the resources for the online course, an average of 83% across the seven statements agreed or strongly agreed. A number of videos, featuring a student who is blind interacting with the WebCT environment were particularly appreciated: The videos of the blind student and the practical work with assistive software are moving experiences for me personally. 286 Australian Journal of Educational Technology, 2003, 19(3) The video was excellent and the gut wrenching experience of listening to a screen reader painstakingly going through absolutely everything was also a highlight. Participants also noted the importance of up to date and relevant resources: The resource material was excellent as it provided opportunities to access “good” sources and useful links. It appears that the combination of videos demonstrating the use of assistive technology (produced in house with Darren Fittler, a UNSW student who is blind), extensive, relevant and current links to resources, and practical activities combined to make a successful environment. There were some issues, however, relating to the fact that the course had initially been designed for both staff development and for students studying the course for credit, which resulted in some confusion: The difficulties I experienced in understanding the two sets of instructions depending on whether you were doing the course for staff development or for credit. Once I worked out this the instructions for the individual tasks were quite clear and easy to understand. This problem was resolved for ITET 3 (as mentioned earlier) by the removal of references to the course for credit. Delivery strategies Five statements were designed based on the Herrington et al checklist to determine the quality and clarity of expectations of the participants, opportunities for dialogue and feedback and whether the tasks could be carried out in the allocated time (this latter item is not specifically included in Herrington’s checklist): • There are sufficient opportunities for feedback. • Course information and expectations are clear. • Instructions for tasks are clear. • The course encourages dialog between you and your colleagues. • There was sufficient time to carry out the tasks and assessment. There was more variation in the responses in this category. While the respondents agreed or strongly agreed that there were sufficient opportunities for feedback and that dialog was encouraged, they were less positive about the clarity of the expectations and instructions. Only 56% agreed with these statements, this was partly due to the problem with ITET 2 of the dual purpose nature of the course, which was rectified for ITET 3. However, there were still some problems for ITET 3: Pearson and Koppi 287 It was hard at times to figure out the order and nature of each task, as there was a lot to cover, although it was all there somewhere. It was a bit hard for me as some of the technical language was unfamiliar i.e. the computer language. I didn’t want to look stupid and hold up the proceedings. I found the last activity a little confusing. The major problem with the course came in response to the statement “There was sufficient time to carry out the tasks and assessment”. Only 19% of respondents agreed with this statement. Although the fellows had clearly enjoyed the module and found it a rewarding experience, many of the comments related to the amount of time they needed to commit to it. It would be helpful to know much further in advance how much time is needed to commit to this module. I would have liked more time to do the other activities because they were good and informative, and it is a shame that there is never enough time. It may be that the extent of the activities and the amount that the participants can expect to do in the timeframe will need to be reconsidered. However, the Fellows were not complaining so much about the amount of work each activity involved, rather that they would prefer to have longer in which to do it, so that they could explore more of the activities. It would be preferable to do the whole module over however many weeks it takes. It was intensive, but very valuable and necessary. It is worth noting that many of the Fellows attended additional voluntary workshops after the official end of the module to get more experience with the assistive technologies, checking tools and accessible document design. Also Fellows from previous cohorts requested follow up ‘revision’ sessions when they responded to the email survey. The amount of time allowed for the module to enable the Fellows to undertake more of the activities is something that will need to be considered. It was clear from the feedback that the participants appreciate the opportunity to actually use some of the technologies and software. One solution may be to redesign the activities so that the whole group participate in at least one hands on activity and one research activity and it is held over a longer period to facilitate that. 288 Australian Journal of Educational Technology, 2003, 19(3) Post-course survey Summative evaluation can be used to assess the changes in performance or attitude that occur as a result of the participants’ engagement with the course (Lackee, 2002). A post-course survey was carried out to determine any change to staffs’ attitude or practice in relation to inclusive course design. The purpose was to discover whether staff regard accessibility as relevant to them in practice as course developers and teachers, and if they have been able to put into practice what they learned during the module. The survey was sent by email to the ITET 1 and 2 cohorts who had taken part in the module six months and twelve months previously. The email simply asked; Has what you learned in the Accessible Course Design part of the program actually made any difference to your design of online courses? Has it had any lasting effect in practice? There was an encouraging response rate with replies from eleven of the thirty Fellows within two days of sending the email. The most frequently repeated comment (seven of the eleven respondents) indicated that the module had resulted in increased awareness of accessibility issues that had not existed beforehand. A typical response was: I think the raising of awareness has been the most lasting consequence. I know now that there is an issue and that it is possible to fix it. Several reported that they now employ at least the basic accessible design features such as ALT tags and the use of checking tools: I incorporate some of what I learned such as tags for figures, images etc. I try to use high contrast colours and provide multiple formats and small files. Although more aware of the issues some have not addressed it in their online courses for a variety of reasons. One, although “sensitised to the issues”, commented: I cannot see a need for it right now. Another felt that more training was required and would appreciate spending more time exploring the creation of accessible documents. A third commented that accessibility was not really relevant to his discipline which is largely laboratory based. Pearson and Koppi 289 One respondent argued that time pressure was a constraint and that as a consequence he takes a “post-responsive” approach: I do not take action before I design my online courses, rather, I take action after the course is created and once I am notified I have a student with disabilities. There was evidence of dissemination through staff support: I have a better awareness of supporting applications for students with disabilities to access IT based information and online. I am able to converse with my support staff about these issues and apply the principles of good instructional design for able and disabled users in hard copy and virtual copy resources that I develop. In summary, the evidence suggests that the accessibility component produced a lasting awareness of the issues involved, that most though not all are at least incorporating the ‘basic’ tenets of accessibility in the design of their online courses, that more emphasis could have been placed on workshops for developing accessible documents, that a few are addressing accessibility in a more comprehensive way in their professional practice and that a minority still regard it as irrelevant to them. The general tone of the responses though indicated that the accessibility module had been a valuable experience: I think the accessible course design part was extremely important as especially when people are starting developing on line, it is so easy to get caught up in the new technologies and all the whiz bang things they can do, and they forget that it may not be accessible to everyone. Although even though I was more experienced in developing online I still found that it was especially good as although I was aware of many of the issues, I easily just forget to take accessibility issues into account but since doing the course and experiencing things first hand I’m far more likely to consider these when developing new material. Recommendations A number of issues emerged from the evaluation of this mixed mode course, some applicable to online course design generally, others that are specific to a staff development course in accessible course design. Clearly the participants found it an interesting and rewarding experience and it has affected to varying degrees their attitude towards inclusive design of courses and resources. • Choose one medium for communication and stick to it – participants want to know that they will find information in one place and don’t want to look elsewhere. Avoid repeating information in email and in the learning environment – the participants become confused as to the 290 Australian Journal of Educational Technology, 2003, 19(3) consistency of the information and where they should look for it. This might be thought contrary to the advice of providing the same information in different ways for novice learners (eg., Salmon, 2000) but what was problematic in this case was that the same information was given inside and outside the learning environment. • Precise and clear communication is essential; one cannot assume the participants will know what you mean, even though they are academics themselves. Academics, IT and library staff have varying degrees of technical competency and that must be taken into account in the language of the course. • Although this was a staff development course and the participants were not assessed, it was useful to have some checklist for learning against which they could measure their own development. • Accessibility issues need to be introduced early enough in a staff development course to encourage and enable inclusive practices to be embraced in a systematic fashion in their own projects. Hands on experience of assistive technologies and checking tools is especially valuable in raising awareness of accessibility issues. • Provide an induction session prior to the module to introduce some of the assistive technologies and checking tools to allow sufficient time for participants to have practice and experience in their use. Also more hands on practice with these technologies during the activities would improve the experience for all the participants. • Most Fellows came for voluntary supplementary sessions after the intensive week was over and follow up requests for more training from previous Fellows suggests a revision session is also beneficial. • Access to the online resources after the module enables staff to keep abreast of developments and encourage continued awareness. Summary The evaluations show that academic, technical and library staff find a practical and theoretical module introducing them to issues of accessibility and inclusion engaging, interesting and relevant. They also show that staff are able to contextualise what they learn about inclusion to their own projects and courses, and that it has a lasting impact on their practice as teachers and staff developers. Staff are willing to use what they have learned and they will at least apply minimum standards. Though a minority will reject change as irrelevant, they are at least cognisant of the issues. Pearson and Koppi 291 Principles of universal design and following guidelines also helped in thinking about our own course designs and the usefulness of a well designed course for all participants. Although we thought we had presented each activity in usable chunks with links to all relevant resources, there was still some difficulty for some people in working out the nature of the task. We will need to carefully revise the activities to try to improve the clarity of the tasks. We tried to avoid the use of technical language as far as possible, but we will also need to carefully consider the way we present some of the technical information, taking account of the fact that the Fellows technical literacy is very varied. Finally, the Fellows are drawn from all sections of the University and include academic, technical and librarian staff All found the accessibility module equally valuable and relevant to their own professional interests whether that be the development of online courses, the provision of library resources to staff and students, or technical support. References Alexander, D. (2002). An accessibility audit of WebCT. Proceedings of The Eighth Australian World Wide Web Conference,(AUSWEB), Queensland, Australia. [viewed 24 Apr 2003] http://ausweb.scu.edu.au/aw02/papers/refereed/alexander/paper.html Alley, L. R. and Jansak, K. E. (2000). Ten keys to quality assurance and assessment in online learning. Journal of Interactive Instruction Development, 14(3). A-Prompt Web Accessibility Verifier. Adaptive Technology Resource Centre, University of Toronto [viewed 8 May 2003] http://aprompt.snow.utoronto.ca/index.html Barker, K. (2001). Quality guidelines for online education and training. [viewed 9 May 2003] http://futured.com/form/pdf/english.pdf Beck, S. (1997). The good, the bad and the ugly: Why it’s a good idea to evaluate web sources. Institute for Technology-Assisted Learning, New Mexico State University, [viewed 9 May 2003] http://lib.nmsu.edu/instruction/eval.html Bobby. Watchfire Corporation [viewed 9 May 2003] http://bobby.watchfire.com/bobby/html/en/index.jsp Brandt, B. L., Farmer, J.A. and Buckmaster, A. (1993). Cognitive apprenticeship approach to helping adults learn. In D. D. Flannery (Ed), Applying cognitive learning theory to adult learning, pp 69-78. Jossey-Bass, San Francisco. Bunker, A. and Vardi, I. (2001). Why use the online environment with face-to-face students? Insights from early adopters. Proceedings of ASCILITE, Melbourne, Australia, pp 111-116. http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/bunkera.pdf Collins, A. (1988). Cognitive apprenticeship and instructional technology: Technical report. Cambridge, MA. 292 Australian Journal of Educational Technology, 2003, 19(3) Forsyth, R. (2001). Participation in online staff development: Why is there a mismatch between intention and practice. Open and Distance Learning Association of Australia 15th Biennial Forum, 24–27 Sept pp 57–58. Grabinger, S. and Dunlap, J. (2002). Applying the REAL model to Web-based instruction: An overview. Proceedings of EDMEDIA, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, USA. Herrington, A., Herrington, J., Oliver, R., Stoney, S. and Willis, J. (2001). Quality guidelines for online courses: The development of an instrument to audit online courses. Proceedings of ASCILITE, Melbourne, Australia, pp 263-270. http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/herringtona.pdf Koppi, T. and Pearson, E. (2002). Design and development of a flexible online course for making accessible online courses. Proceedings of EDMEDIA, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, USA, pp 1019-1031. Lockee, B., Moore, M. and Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, (1), 20-26. http://www.educause.edu/ir/library/pdf/eqm0213.pdf McAlpine, I., Koppi, A. J., McLean, J., Hodgson, L., Fardouly, N. and Kinch, S. (2001). Teaching quality principles and guidelines. Educational Development and Technology Centre, University of New South Wales [viewed 9 May 2003] http://www.edtec.unsw.edu.au/ Nielson, J. (2000). Why you only need to test with 5 users. Jakob Nielsen's Alertbox, March 19 [viewed 9 May 2003] http://www.useit.com/alertbox/20000319.html Owston, R. D. (2000). Evaluating web-based learning environments: Strategies and insights. Cyber Psychology and Behavior, 3(1), 79–87. Patton, M. Q. (2002). Qualitative Research and Evaluation Methods (3rd Ed) SAGE Publications. Pearson, E. J. and Koppi, A.J. (2001). Guidelines For Accessible Online Courses. Internal publication, University of New South Wales. http://www.edtec.unsw.edu.au/intra/files/accessibilty/default.html Pearson, E. and Koppi, T. (2002). Essential elements in the design and development of inclusive online courses. Proceeding of EDMEDIA, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, USA, pp 1569-1575. Salmon, G. (2000). E-Moderating: The key to teaching and learning online. London, Sterling (USA): Kogan Page. Wave 3.0 Accessibility Tool, WebAim, Utah State University [viewed 9 May 2003] http://wave.webaim.org/index.jsp Elaine Pearson, University of Teesside, Middlesbrough, UK. Email: e.pearson@tees.ac.uk Associate Professor Tony Koppi, Director, Educational Development and Technology Centre (EDTeC), University of New South Wales, Sydney, Australia. Email: t.koppi@unsw.edu.au