Microsoft Word - gosper-v4.doc Australasian Journal of Educational Technology 2007, 23(2), 227-247 Selecting ICT based solutions for quality learning and sustainable practice Maree Gosper, Karen Woo, Helen Muir, Christine Dudley and Kayo Nakazawa Macquarie University This paper reports on a project involving software selection in the context of a curriculum redesign of a university level Japanese language program. The project aimed to improve learning outcomes, increase flexibility in student access, and increase flexibility in approaches to teaching and learning, through the use of a variety of software packages and digital resources. In doing so, an imperative was to ensure the solutions adopted were manageable within the existing organisational arrangements of the Department and the University. The selection process has led to the development of three instruments which form the CICTO Framework for Software Selection. Introduction Introducing changes to the curriculum is always challenging. This is especially true when the changes involve the integration of information and communication technology (ICT) based solutions into the curriculum. Such changes have the potential to impact on learning, academic practice and organisational arrangements. This paper reports on the process of software selection for curriculum change. The context in which the curriculum change was situated was a project to convert a program in Japanese language to flexible delivery mode. Prior to the project, the Japanese Studies staff had acquired considerable experience with teaching in an e-learning environment, which had led to an understanding of the potential of ICTs to provide multimedia content, communication opportunities and interactive learning experiences. They were also aware that the recent generation of social and communication software solutions and the availability of high quality interactive language resources on the Internet had the potential to enhance their existing ICT based programs, and provide more sophisticated verbal interactions, collaboration and feedback relevant to the language learning context. Thus, 228 Australasian Journal of Educational Technology, 2007, 23(2) the need to review existing approaches and evaluate new ones was a major impetus for the project. Providing a more flexible learning environment for students was another driver. As well as balancing work, family and lifestyle commitments, an increasing number of students are enrolling in the language program concurrently with studies in other disciplines, or as part of a double major program. These students could benefit greatly from more flexible timetabling and more flexible access to resources and learning experiences. Changing financial arrangements within the Department, combined with increasing demands on time, space and resources, also contributed to a need to explore ways of increasing the effectiveness and efficiency of the existing teaching model based on a traditional campus based experience. A reduced dependency on face to face teaching through the effective use of technology was seen as a way to provide more flexibility in teaching and learning, greater access to resources, and a reduction in demands on classroom spaces. Preliminary explorations of possible solutions, prior to the onset of the project, led to the realisation that one software solution would not fit all, due to differences in approaches to teaching, learning processes and learning outcomes. Hence, a suite of solutions from which teachers could choose was desirable. There was also the realisation that the effectiveness of any ICT based solution could be completely undermined by a lack of fit with the organisational arrangements surrounding its use, for example, a lack of interoperability with existing technical infrastructure and administrative procedures, or inadequate training and support for users. Collectively, this pointed to the need for a software selection procedure that focused primarily on curriculum imperatives and the needs of the learner, but at the same time addressed the technical capability of the software and its fit within the organisational environment in which it was to be used. Following is an overview of the procedure adopted, the instruments of analysis that were developed and the experiences encountered in the quest for sustainable solutions. Current software selection procedures When reviewing the literature on frameworks to select and evaluate educational software, we had two specific requirements. The first was in relation to the scope of the framework - we needed a single comprehensive framework that integrated the educational, technical and organisational aspects of software selection. The second was in relation to the perspective adopted - we needed a framework for an audience that did not have a comprehensive understanding of the complexities of technical specifications and standards, but were, nevertheless, responsible for Gosper, Woo, Muir, Dudley and Nakazawa 229 ensuring the quality of the educational experience. The framework needed to provide sufficient detail to alert this audience to issues that may not be immediately apparent, but which would impact on the efficiency and effectiveness of teaching and learning. Once identified, these issues could then be referred to those with relevant expertise. After reviewing the extensive body of literature on software selection, we found three frameworks that had elements suiting our needs. In the context of educational change, we agreed with the proponents of activity theory who suggest that successful educational change does not happen merely by introducing technology. It is dependent on several factors: the psychology of teachers, technical support, ease of access to technology, and institutional support (Ringstaff, Kelley & Dwyer, 1993, cited in Bellamy, 1996). The same concerns were echoed by Collis and Moonen (2001), who proposed a 4-E Model to describe the factors that influence adoption of technologies for teaching and learning. The successful uptake of technologies by an individual is a function of the four Es - the environment (organisational context), educational effectiveness, ease of use, and engagement (personal engagement of teachers). Working from the teacher’s perspective, Bates and Poole (2003) developed their SECTION model for software selection which advocated the following for consideration: • Students and the appropriateness of the technology for their needs and circumstances • Ease of use and reliability of the technology • Costs for institutions, staff and students • Teaching and learning contexts, and the capacity of the technology to support identified activities and processes • Interactivity which can be enabled through the technology • Organisational requirements and the changes that need to be made to the organisation and the technology to make it work • Novelty factor associated with the technology • Speed of use, including time for courses to be developed and materials changed. Based loosely on a combination of the three frameworks, we have developed three analysis instruments to assess software suitability within a given educational, technical and organisational context: 1. a curriculum needs analysis 2. an ICT requirements and capability analysis 3. an organisational support and interoperability analysis. 230 Australasian Journal of Educational Technology, 2007, 23(2) The software selection procedure in action The software selection procedure begins with the curriculum needs analysis. At the conclusion of the analysis, one or more software packages may emerge as potential solutions for supporting specific teaching and learning processes. These software packages are then subjected to an analysis of their technical capabilities in relation to defined needs and their fit with the organisational environment in which they are to be used. The entire procedure is iterative, with continual reference back to the identified educational needs to ensure that educational efficacy is maintained. The curriculum needs analysis An advantage of fully analysing educational needs before exploring software solutions is that a clear understanding of the curriculum requirements can be identified and stabilised, in the dynamic context of software innovation and development. Where ICT based solutions are based primarily on software capability rather than educational needs, there is the danger of the curriculum becoming redundant as the software becomes outdated. Hence, the aim of the curriculum needs analysis is to clearly define the educational context in order to make informed decisions about the functionality required of a software solution. Five categories of issues, which reflect the essence of the pedagogical dimensions discussed by Reeves (1997), were identified as being important in defining this functionality. The categories, along with the key issues to be explored in each are shown in Figure 1. The first category addresses the learning environment. Bates and Poole (2003) note that “… the choice and use of technology are absolutely dependent on beliefs and assumptions we have about the nature of knowledge, how our subject discipline should be taught, and how students learn” (p. 25). Different philosophical perspectives engender particular pedagogical approaches which, in turn, influence the roles and expectations of teachers and students (Bates & Poole, 2003; Gillani, 2003; Reeves, 1997). Software, for example, that is designed for highly structured, instructor led environments with well defined pathways through activities, may not be suited to an environment which aims to support co-constructed experiences where students can determine pathways and activities, and actively communicate and collaborate in pursuit of knowledge and understanding. Hence, it is important to have a clear understanding of philosophical and theoretical underpinnings to ensure that there is compatibility with the chosen software. The second category explores the nature of the student cohort. Students come with great diversity in their background and experiences. Different Gosper, Woo, Muir, Dudley and Nakazawa 231 1. The learning environment: Philosophical and theoretical underpinnings • What philosophical and theoretical perspectives underpin teaching and learning in your discipline and/or the course being analysed? • How would you define your role as teacher? • What expectations do you have of your students as learners? • What implications do the above have for the learning environment you are creating? • What do you see as the main role for technologies in supporting this environment? 2. The student cohort • What types of diversity are you aware of that will need to be catered for in the expected student cohort e.g., prior knowledge and skills, age, access to technology, IT skills level? • Are you aware of any specific learning needs present in the student cohort? • What is the enrolment mode of your students ( e.g., on campus, distance students or international) and what implications will different modes have for the delivery of learning experiences (e.g., face to face or online)? • What implications does the learner profile have for the design and delivery of the curriculum, as well as the technical solutions that could be considered? 3. Aims and outcomes • What are the teaching aims and the related learning outcomes for this course/unit? • What knowledge and skills and attitudes will be developed? • What content will be used to exemplify the defined knowledge skills and attitudes? 4. The teaching and learning processes, activities and resources • What learning activities and processes are necessary to support the achievement of the nominated outcomes? What will the learner do? • What teaching methods and strategies are employed to facilitate the identified activities and processes? What will the teacher do? • What resources are needed to support the teaching and learning activities defined above? • Can these resources be sourced from existing repositories (e.g. libraries, object repositories, the Internet)? • How can technologies support the defined teaching and learning activities? What functionality is required of technologies? What software solutions are available? 5. Assessment and feedback • How will each of the defined outcomes be assessed? • How will feedback be provided for summative and formative tasks? • How can technology support the assessment and feedback process? What functionality is required? What are the possible software solutions? Figure 1: The curriculum needs analysis 232 Australasian Journal of Educational Technology, 2007, 23(2) levels of prior knowledge and experience in a particular subject will influence the design of resources and activities, and the type of scaffolding needed to support learning. Cultural background, IT skills, access arrangements and modes of study will all influence the choice of technology as well as the support required to ensure the effective and efficient use of the technologies for learning. In addition, age and experience in the digital world will influence students’ attitudes towards and expectations of the use of technologies for teaching and learning (Oblinger & Oblinger, 2005). We therefore need to understand the characteristics of the student cohort to ensure that the technology chosen is appropriate and will not impede their learning. In categories 3 and 4, we clearly define aims, outcomes, teaching and learning processes, and the supporting activities and resources. It is of fundamental importance that these elements are aligned (Biggs, 2003). Teaching strategies, learning processes, activities and resources are invariably different for different teaching aims and learning outcomes (McKeague & Di Vesta, 1996). In the Japanese language context, for example, the processes underpinning the aims and outcomes associated with the development of vocabulary are quite different to those for the development of communication skills and increasing cultural awareness. Category 5 addresses assessment and feedback. As we know, assessment has a pivotal role in motivating students to learn, in providing feedback on progress and in providing measures of achievement. Technological solutions can be employed in various ways to support the assessment process, for example through quizzes, simulations and modeling exercises. Timely and rich feedback can be generated through discussion forums and the use of rubrics. In addition, software packages often come with in-built monitoring tools to review student’s progress (Macdonald, J. 2002; Phillips & Lowe, 2003). The choice of software will depend on how each of the defined outcomes is assessed, and how feedback is to be provided for both formative and summative tasks. When applying the curriculum needs analysis, we shared the same experience as Burston (2003), who found that the most difficult aspect of software selection was establishing the “teacher fit”, because teachers find it hard to articulate the theoretical underpinnings of their practice. In view of this, and to assist the process, the analysis was facilitated by an academic developer, with expertise in curriculum design and development. The facilitator guided a series of discussions that enabled the Japanese Studies team to analyse their teaching and learning context using their discipline specific terminology. Gosper, Woo, Muir, Dudley and Nakazawa 233 The Japanese Studies team had previously developed a completely online Japanese beginners program named Gengoro which had been integrated into various other ICT language teaching projects. They had already collected a substantial amount of relevant information and knowledge of ICTs and had a sound understanding of the technology issues relating to the diversity in their student cohort. For them, the more challenging aspect of the needs analysis was articulating their philosophy and unpacking the contributions technologies could make to the curriculum. Nonetheless, on reflection, the team noted that one of the key lessons learnt was that articulating the teaching philosophy was central to understanding educational needs, for it is educational need that determines the criteria for selecting the software. Embedded in their philosophy was the primacy of the role of the teacher in providing the best possible environment for students to learn. This involves providing high quality learning materials, motivating learning experiences where students are comfortable to explore and take risks, and providing opportunities to build on their prior understandings. It also involves providing students with feedback as well as opportunities to assess their own progress. While the learning process is seen as a joint endeavour by the teachers and students, it is the responsibility of students to take advantage of all of the opportunities provided. Good two way communication between teachers and students and collaboration amongst participants is important to achieving successful outcomes. Technology to support this philosophy needed to include effective individual and group communication tools, access to authentic (online) learning materials, and facilities interacting with content and exploring ideas. Technologies that allowed self paced learning and catered well for student feedback, and automatic or delayed teacher responses were also important. To explore issues relating to curriculum, we began by identifying the ‘key aspects of language learning’, namely writing, speaking, listening, and cultural awareness. Although described as separate entities, a holistic approach to teaching was adopted to reflect the implicit interrelationships evident in proficient Japanese speakers. Addressing these aspects involved recognition of vocabulary, reproduction and imitation of vocabulary, knowledge and application of grammar, deconstruction of words and sentences, intercultural awareness and understanding, discrimination in language use, as well as spoken and written production of language. These were the elements for which we were seeking support through technical solutions. In addition, provision of effective and efficient feedback to students as they were developing their written and oral communication skills was another area for solutions. 234 Australasian Journal of Educational Technology, 2007, 23(2) Table 1: Curriculum alignment grid Teaching aims Learning Outcomes Learning processes Instructional processes and learning experiences Instructional resources Possible software solutions Recognition of characters and scripts (Hiragana Katakana and Kanji) Being proficient in reading and understand- ing simple passages using Japanese scripts Memorising Japanese characters Decoding words and their meaning both in and out of context Comprehend- ing vocabulary and grammar Classroom instruction using examples of scripts and practicing for recall Use of dictionary to make meaning of the words Flash cards, computer based exercises Written scripts and readers Kantaro CD Online quizzes with feedback Development of intercultural awareness and understand- ing Being able to communicate in written and spoken forms in a way that shows an awareness of norms, cultural/ social differences. Perceiving different contexts Analysing different behaviours Discrimina- ting between different behaviours in different contexts Understand- Ing cause and effect in behaviour Developing self reflective strategies Use of exemplars Exposure to authentic experiences Modeling by teachers Communica- ting with people from different cultures Role playing Videos Broadcast TV Newspapers Web sites Books etc Online discussions, email and chat to enable interaction with authentic speakers Student presentations (e.g., PPT ) Use of Wikis for group projects Use of Blogs for personal reflections Online role plays To ensure the technologies proposed were closely aligned with the intent of the curriculum, a simple alignment grid was used to articulate the relationships between teaching aims, learning outcomes, learning processes, instructional processes, learning experiences and resources. Two examples showing the alignment process in action are presented in Table 1. The examples are for aims associated with recognising characters, and developing cultural awareness and understanding. Use of the grid also had the added advantage of assisting teachers to articulate a rationale for their practice. An extension to the grid, not shown in Table 1, is a column for assessment tasks to ensure their alignment with all the elements of the curriculum. Arising from the curriculum needs analysis, several software solutions were identified as being suitable. For example, Wimba, software to enable Gosper, Woo, Muir, Dudley and Nakazawa 235 classroom collaboration and interactive communications, with voice facilities to support speaking and listening exercises, was particularly suitable for recognition of vocabulary, reproduction and imitation of vocabulary, and spoken production of language. The use of videos, broadcast TV, newspapers, and web resources that exemplified authentic contexts were particularly suited to the development of intercultural awareness and discrimination in language use. Text based discussion tools and quizzes available in WebCT (a learning management system) and LAMS (a system for sequencing learning activities) could be employed to develop activities for the deconstruction of words and sentences and written production of language. Each had strengths and weaknesses in relation to educational needs. To develop a more comprehensive understanding of the impact of these technologies on staff and students from educational and operational perspectives, each of the solutions was subjected to further analyses using the ICT requirements and capability analysis, and the organisational support and interoperability analysis. ICT requirements and capability analysis Technical requirements have been discussed in some detail in the literature (e.g. Foshay and Almed, 2000) and are often presented as checklist for rating capability. In our experience we found that checklists were insufficient to make decisions about the adoption or rejection of a particular software because they cannot capture the complexity of the technical environment. The critique of evaluation checklists offered by Tergan (1998) mirrored our own concerns about their use. First, evaluation criteria are often ambiguous and inter-rater consistency is low. Second, there has been little evidence provided on the validity of the criteria used in these checklists. Third, checklists often result in a single score, which is problematic because “simply counting the ratings of the criteria items of the particular category may distort the result of the evaluation of that particular software aspect.” Lastly, Tergan pointed out that evaluation criteria are based on learning philosophies that are not articulated, and that variations in educational needs in different teaching contexts are often neglected. As a result we realised we required an instrument that could raise awareness of the underlying technical issues related to integrating the software into existing technical infrastructure. The instrument also needed to be focused less on the technical minutia - IT experts could do this - and more on the capability of the software to accommodate the administrative, teaching and learning functions and workflow patterns arising from the requirements identified in the curriculum needs analysis. Too often the educational limitations of the software only become evident after substantial commitment has been made in terms of dollars and time. 236 Australasian Journal of Educational Technology, 2007, 23(2) Complicated, time consuming and costly adaptations to overcome the limitations can render the software unsuitable, despite a sound educational fit. Therefore, the instrument needed to guide teachers in evaluating the potential limitations and benefits of software solutions in light of the identified teaching and learning processes. As a result of our explorations, we adapted a checklist, developed by Foshay and Almed (2000), of the technical requirements for a range of educational software solutions. The outcome, the ICT requirements and capability analysis, is a questionnaire suited to teasing out the complexities present in a large networked environment. Throughout its development we were conscious that the functionality required of a piece of software is context specific. Furthermore, although we have attempted to define some of the functionality that may be required, we are aware that a definitive description is not possible given the dynamic nature of the software environment and the needs arising from different educational contexts. For our purposes, we found six broad categories of issues to be of particular relevance. The categories along with the issues to be explored are shown in Figure 2. The first category, teaching and learning process management, serves the purpose of ascertaining the strengths and weaknesses the software in question has in supporting the processes identified in the needs analysis. In reality this was an iterative process - explorations of the capability of the software led to new insights and ideas about teaching which, in turn, led to changes to the original conceptions of the desired teaching and learning processes and interactions. The second category looks at issues arising from assessment requirements. One of the requirements for Japanese studies was the provision of efficient and effective feedback to students when developing vocabulary and related declarative knowledge. The use of computer based applications for this purpose has been well recognised (Fletcher-Flinn & Gravatt, 1995; Kulik & Kulik, 1991). The oral capability of Wimba offered potential solutions for the development and assessment of written and spoken vocabulary and speaking skills. However, associated with the use of technologies for assessment purposes were issues related to security, authentication and data management which placed additional demands on the functionality required of the software. These and similar issues needed to be identified and resolved for the software to be efficient and effective. The management of content, as addressed in the third category, has great potential to reduce the workload of teachers, especially in relation to reusing content from semester to semester, or transferring from one program to another. The ability for teaching staff to be able to easily Gosper, Woo, Muir, Dudley and Nakazawa 237 develop materials themselves was of particular importance to the Japanese Studies team, not only for maintaining currency, but also to reduce on- going specialist development costs. 1. Teaching and learning process management In the curriculum needs analysis you identified the educational functionality required of this piece of software in relation to teaching aims and learning outcomes, and more specifically: • The teaching processes / activities it needs to facilitate • The learning processes / activities it needs to facilitate In view of this, what are the strengths and weaknesses of the software package. 2. Assessment If the software is to be used for assessment purposes, consider the strengths and weaknesses of the software package in regard to the following features: • Storage of completed assessment tasks for review by staff • Security of assessment data and ability to define access levels for different users • Ability to collate results and store results • Ability to export results to other administrative systems • Ability to vary feedback to students in relation to timing of the feedback and the extent of feedback given • Ability to provide marks and progress information to students. 3. Content creation and management The content creation and management features required of the software will be dependent on the defined educational requirements specified in the needs analysis. In relation to the educational requirements: • What content management and creation features are required? • Which of these features are supported by the software? • Which of these features are not currently supported? These may include: • Creation of content by students and/or staff • Importing and exporting content (e.g. multimedia content, spreadsheets) • Customising and editing content • Managing and re-using content (e.g. duplicating files, moving content across modules) • Storing content • Including multimedia content (video, sound) • Supporting foreign language (display/recording and/ or output/input). 4. User management User management features may include the provision of tools or functionalities. In relation to the educational requirements, which of the following features are required? If so, are the features currently supported by the software? • Managing and controlling access to resources by various types and groups of users 238 Australasian Journal of Educational Technology, 2007, 23(2) • Providing sufficient capacity to accommodate the expected number of users, classes, instructors and records • Enabling learners /users to save work, exit, and resume work at that point later on • Tracking students performance (e.g. personal reports, summary tables, charts, and graphs, describing response patterns and score/ grade/ mastery obtained, basic statistics of tests) • Monitoring students processes (access, progress, length of completion) • Providing adequate levels of security for handling student information. 5. Technical usability There are standard technical and usability features that must be operating at an acceptable level before software can be considered for educational use. IT staff will have their own checklist to ensure the software package in question meets institutional requirements. Teaching staff need to consider their satisfaction with the following features: • The reliability of the software package (stable and error-free) • The speed of the software package when run on existing lab computers • The speed of the software package when used over a network by remote students • Backup and disaster-recovery capability. • Ease of navigation for students and staff • Availability of documentation and in-built help functions for staff and students • Access to technical support for staff and students. In relation to the educational requirements and the capabilities of students and staff: • Can the software work effectively with the existing technologies (network, hardware, platform, peripherals, browsers, plug-ins and complementary software)? • If the software is expected to be used by students on their own computers, are the technologies required to operate the software available to all (present and future) students (network, hardware, platform, peripherals, browsers, plug-ins and complementary software)? • Are additional technologies required for it to operate effectively? • Are there implications the additional technologies may have (in terms of cost, maintenance, etc.)? 6. Future growth Does the software package have potential to meet a range of current and future teaching and learning needs? Figure 2: ICT requirements and capability analysis Another aspect for consideration was in relation to content creation. First, different content presentation modes (visual, audio, multimedia) may desirable but not all software programs support multimedia content. Second, the role of students in relation to content creation has to be Gosper, Woo, Muir, Dudley and Nakazawa 239 considered. In the past, much of the content being used for teaching has been provided for students to access (Miodusa & Nachmias, 2002). However, as we move from a teaching paradigm of acquisition of information to co-construction of learning (de Boer & Collis, 2002), there is the need to enable students to create content individually and collaboratively. This was a requirement for the Japanese Studies team when developing intercultural awareness. These and other considerations in relation to content creation and management are listed in category 3. Managing users is the fourth category. Providing access, creating and managing student and staff accounts and passwords can be time consuming. Some software packages may be interoperable with the institution’s student systems to allow direct feed of student data and hence saving administrative work. There were also a number of user management features that were desirable for the Japanese Studies context, for example, grouping and defining access for student groups, providing individualised pathways and options and enabling individuals to save work, exit and resume at a later point. In some cases there was the need to track performance and monitor student progress. Some of these features may be built into the software; others may not, therefore it is important to identify need from the outset. Collis and Moonen (2001) found that efficiency and ease of use were important conditions for the successful adoption of software. Reliability of performance, security of data, easy access, fast download speeds, intuitive navigation, and availability of help functions are some of the recognised usability features (Bates & Poole, 2003; Burston, 2003; Foshay & Ahmed, 2000; Krauthamer, 2000) that needed to be considered for the smooth operation and the acceptance of the software by all its users. The technical usability issues of this nature relevant to the Japanese Studies context are listed in category 5. Finally, because of the cost, time and effort associated with integrating new technologies into the curriculum, it was desirable that the software met a range of teaching and learning needs rather than having limited applications. Therefore, the potential of the software to meet future need was an important consideration. When undertaking the ICT requirements and capability analysis, the Japanese Studies team was not always able to answer the technical questions posed and assistance was sought from the relevant support staff on campus. The ensuing discussions were informative but also exhausting and there were moments when the team felt intimidated by the technical terms used in the discussion. Too much detail in some instances also proved to be confusing and some of the questions could only be answered after the software had 240 Australasian Journal of Educational Technology, 2007, 23(2) been trialed. Nonetheless, the overall process of considering the usage of the software in depth was necessary for building a sophisticated understanding of the requirements and the software’s capability in meeting those requirements. In practice, many decisions required a careful weighing up of strengths and weaknesses. Wimba, for example, was proposed as a possible solution to providing interactive speaking and listening opportunities for the students. Unfortunately a current limitation of Wimba was the version in use did not allow Japanese script input. This meant students could not experience dual channel input from reading Japanese script and listening to a voice over. Despite this limitation, Wimba could still be used to provide voice over combined with English language script. In addition it was established that the software worked well for on campus access and for most students at home. It was therefore decided that the advantages of the package far outweighed the disadvantages. Organisational support and interoperability analysis The introduction of new technologies has implications for the whole educational enterprise including infrastructure arrangements, curriculum development, teaching and learning practice, support for staff and students, academic policy and practices. Therefore, the final step in the analysis was to check the fit with the organisational context within which the software was to be used. Six areas of particular relevance to the Japanese Studies context were identified for inclusion in the organisational support and interoperability analysis: software administration and interoperability with enterprise systems; student and staff training and support; financial and asset management arrangements; compliance with legislative requirements; alignment with academic practice, policy and procedural frameworks; and provision of risk management. These are outlined in Figure 3, along with the issues for exploration under each of the categories. In relation to these issues, adopting software that was supported by the University’s Central Services had major advantages for Japaneses Studies staff and their Department. When the software is supported, as in the case of the University’s learning management system, WebCT, teaching staff are given access to the software and its tools; technical support is provided through a Help Desk; hands on training and documentation is provided for students and staff in how to use the software; user accounts are automatically created and managed. Staff do not have to concern themselves with the back-end technical infrastructure arrangements related to hosting arrangements on servers, backups, authentication of users, Gosper, Woo, Muir, Dudley and Nakazawa 241 1. Software administration and interoperability with enterprise systems • What existing systems does the software have to be integrated with? • To what extent is the software integrated into the required systems; what additional hardware or software is required? • What mechanisms are in place and who is responsible for: o system security, backups, authentication of users? o the installation and monitoring of the software? o the management of software for staff and students (upgrades, licenses)? o the management of user accounts (creation, removal, etc.)? o backing-up student data and access records? • To what extent can these and related tasks be accommodated by existing academic or support staff? 2. Financial and asset management The cost of adopting a piece of software is often not a one-off expenditure. There may be costs related to ongoing maintenance and updates, hardware, physical lab space, etc. • What is the licensing cost and agreement for the software package? • Is there assured continuity in the funding for the software? • Are there an ongoing maintenance costs? • Does the use of the software package incur cost for students? • Does the use of the software require special physical space arrangements? • Are there any other financial or asset implications associated with use of the software? • What mechanisms are in place and who has responsibility for these tasks? • To what extent can these and related tasks be accommodated by existing academic or support staff? 3. Compliance – university and regulatory frameworks • Are there implications for the use of this software in relation to compliance policies and procedures for example: o general accessibility o disability access o intellectual property / copyright o records management o confidentiality / privacy • What mechanisms are in place and who has responsibility for these tasks? • To what extent can these and related tasks be accommodated by existing academic or support staff? 4. Student and staff training and support (software) The training and support needs of the staff and students may be required. • What training, documentation and support is required? • Who is responsible for: o staff and student training and documentation? o help desk support – 24 x 7 • To what extent can the requirements be accommodated by existing academic or support arrangements? 242 Australasian Journal of Educational Technology, 2007, 23(2) 5. Academic practice and arrangements (curriculum and resource development, teaching, administration) Changes to workloads and roles and responsibilities of all staff are likely to occur during the development and implementation of new solutions. • What provision of support (time allocation, resources, personnel) will be needed for academics and support staff in relation to o educational design o preparation of materials o updating materials (per semester) o teaching/ monitoring students o assessing student work • Does use of the software have implications for academic policies and procedures e.g: o assessment o course development and review o quality assurance and enhancement • To what extent can the support be catered for within existing resourcing and staffing arrangements? 6. Risk management • What are the risks of using this in relation to: o technical infrastructure arrangements o administrative and support arrangements o curriculum design and teaching activities o students and their learning o staff workloads • What mechanisms are in place to deal with these risks and who has responsibility? Figure 3: Organisational support and interoperability analysis administration of licenses, software upgrades and patches, risk management strategies, etc. Professional development opportunities and educational development services are available to assist with integrating the software into their curricula and developing resources. Central Services provide guidelines and advice to ensure adherence to the regulatory frameworks that the University has to comply with including those related to accessibility, confidentiality and privacy, intellectual property and copyright, and records management. Financial arrangements are in place to ensure the costs for licenses and upgrades are covered and the financial outlays for students are kept to a minimum. Taking into account arrangements for the above, adopting software that was not centrally supported would have implications for both academic and general staff arising from: • changing workloads, in particular the work involved in teaching online and in the developing innovative approaches to teaching and learning Gosper, Woo, Muir, Dudley and Nakazawa 243 • the provision of support at all stages of the educational developmental cycle: design, development and implementation. • the resourcing of ongoing maintenance and further development of online courses and resources. • changing roles and responsibilities arising from administering, teaching and learning in an e-learning context. If not taken into consideration, these and similar issues have the potential to impose untenable costs and workloads on staff which will ultimately undermine the quality of teaching and learning. The organisational interoperability analysis was complex and input was required from a number of organisational units across the university. Nevertheless there were benefits in undertaking the analysis. Technical staff were able to appreciate the full complexity of teaching and learning needs; likewise academics were able to better understand the limitations of the software and the complexity of the infrastructure arrangements necessary to achieve seamless delivery. There was also the realisation from all parties that selecting software was not an individual act. It required a team effort in uncovering all the educational, technical and organisational ramifications. Pulling it all together The final decision about which software to use became one of balancing educational needs, technology requirements, and organisation inter- operability. In the case of the Japanese Studies program, the organisational support and interoperability analysis became the bottom line for making decisions. If the software was not supported by Central Services, or imposed an unmanageable load on existing staff, the decision was not to use it because of the amount of work and responsibilities involved. After that, it was a question of balancing educational need against technical capability. The individual items on the instruments highlighted the full range of issues and complexity involved in making a decision, but ultimately, we found it useful to reflect on the overall educational efficacy guided by the following questions: • Does the software support a defensible instructional methodology that is aligned with philosophical perspectives of the course? • Can the software be well integrated into the curriculum by supporting instructional strategies and facilitating learning experiences that are aligned with the aims and outcomes of the curriculum? • Does it have appropriate levels of interaction and regulation in learning and representation of subject matter? 244 Australasian Journal of Educational Technology, 2007, 23(2) • Is it efficient and easy to use, i.e. does it make teaching and learning easier to accomplish? • Is it effective, i.e. does it produce better outcomes, or at the very least comparable outcomes? • Is it innovative, i.e. does it allow teachers to do something new? • Does it contain features for motivating students? • What value does it add to the suite of software available to you and your students? • • Do the costs for staff and students associated with using this software outweigh the benefits afforded for teaching and learning? (adapted from Burston, 2003) For the Japanese Studies team, some decisions about the software to be included in their suite of solutions have been made. WebCT, as the learning management systems, scored high on the organisational analysis, and quite high on both the curriculum needs and technical capability analyses. Hence, it will continue to provide the administrative front end for all courses. Much of the course content, including links to external web resources, will continue to be housed within WebCT and extensive use will be made of the built-in assessment and communication tools. The quiz tool will be used for formative assessment and feedback on vocabulary, grammar structures and kanji acquisition. Quizzes will also used for summative assessment purposes. The discussion forum, which supports Japanese language script, is particularly useful for discussions aimed at developing cultural awareness and for supporting group work, group communication and feedback. As well as enabling individual communication with students, the mail tool will be used for assignment submission and the provision of individual feedback. Collectively the functionality provided through WebCT makes for both efficiency and ease of use for staff and students. Wimba scored well in all analyses for its 'Voice Board' feature and will also be included as a tool across the Department. It proved to be easy to use by both students and staff. The software enabled students to listen and prepare spoken responses; practise, edit and review their responses; record responses; listen to other students’ responses and communicate with other group members. For teachers, the software was particularly useful for modeling spoken language, providing spoken feedback in English and Japanese, assessing spoken Japanese and monitoring the progress of individual students. For students, practising language skills using Wimba was often less intimidating and less confrontational than in a face to face environment, and it allowed them to more easily concentrate on their areas of weakness. Overall, the inclusion of W i m b a in the suite of software solutions greatly enhanced the teaching and learning of the spoken component in the curriculum. Gosper, Woo, Muir, Dudley and Nakazawa 245 iLecture, a university wide, web based lecture recording technology, enables the automatic recording of lectures which are then made available to students through a web interface in WebCT. This tool is used to support lesson review and to assist students with special needs. Like WebCT, Wimba and LAMS, iLecture is centrally supported within the University and hence scored high on the organisational analysis. It also scored high on the technical capability analysis and high in some aspects of the curriculum needs analysis in that it caters for students with special needs. It will continue to be used as part of the strategy to deal with classes with a wide range of abilities. LAMS is to be used principally as a management tool to organise multi- level tasks associated with research projects, groupwork and student presentations. Its capacity to support group interaction, link to external web resources and share files is useful for scaffolding project development. From a teaching perspective, the ability to provide structure through the provision of a predefined sequence of tasks and to monitor a student’s progress along the sequence could be useful for supporting the staged development of knowledge and skills. Although it has potential, its strength as defined by the curriculum needs analysis and the technical capability analysis was limited to teaching support and the facilitation of project development processes for students, therefore it will be included as an optional choice for staff. Concluding comments The integration of ICTs into the curriculum brings unique challenges which heighten the critical interrelationships between the technology, the curriculum, and the organisational environment in which they are embedded. The technology has to work: this is a non-negotiable precondition for learning. The technologies being used must serve the needs of the curriculum, not vice versa. They must be effective in supporting the processes of teaching and learning; they must be easy to use; and they must be efficient in the time and mental energy expended in their use. The effectiveness of any ICT based solution could be completely undermined by a lack of fit with the organisational arrangements surrounding its use. Poor interoperability with academic and student support structures; poor alignment with workflow and academic practices; lack of integration into policy and procedural frameworks; and poor IT support and training can lead to frustrations and inefficiencies that are unsustainable for students and staff, and in the long term can lead to poor learning outcomes. Therefore, strategies for selecting software must address all three elements - the curriculum, the technical capability and the organisational environment - in a comprehensive and cohesive manner. When one of these elements is out of balance, the quality of teaching and learning will be severely affected. 246 Australasian Journal of Educational Technology, 2007, 23(2) The three instruments that have been developed for this project are an important step in ensuring a comprehensive and balanced approach to software selection. Collectively they form the CICTO Framework (curriculum, ICT and organisational) for integrating software solutions into educational environments. It must be recognised that the Framework is a living entity and will change over time as the educational context, technologies and organisational arrangements mature. Each context is different, therefore the categories and issues will vary from one context to another. Nevertheless the value of such a framework is that it places the focus of decision making at the heart of the matter – the educational experience. Acknowledgement The teaching development project was supported by the Macquarie University Flagship Scheme. We would like to acknowledge the generous support provided by staff at the Centre for Flexible Learning and the Macquarie E-Learning Centre of Excellence for the trialing and development of software solutions. References Bates, A. W. & Poole , G. (2003). Introductory remarks on knowledge, learning and teaching. In Effective teaching with technology in higher education. San Francisco: Jossey-Bass. Bellamy, R. K. E. (1996). Designing educational technology: Computer-mediated change. In B. A. Nardi (Ed.), Context and consciousness (pp. 123-146). London: MIT Press. Biggs, J. (2003). Teaching for quality learning at university. Philadelphia, SRHE and Open University Press. Burston, J. (2003). Software selection: A primer on sources and evaluation. CALICO Journal, 21(1), 29-44. de Boer, W. & Collis, B. (2002). A changing pedagogy in e-learning: From acquisition to contribution. Journal of Computing in Higher Education, 13(2), 87- 101. Collis, B. & Moonen, J. (2001). Will they use it? In B. Collis & J. Moonen (Eds.), Flexible learning in a digital world: Experiences and expectations (pp. 44-66). London: Kogan Page. Fletcher-Flinn, C. M. & Gravatt, B. (1995). The efficacy of computer assisted instruction (CAI): A meta-analysis. Journal of Computing Research, 12(3), 219-242. Foshay, R. & Ahmed, M. I. (2000). A practical process for reviewing and selecting educational software. PLATO Learning. Inc.. Bloomington. MN. (Report No. BBB37097). (ERIC Document Production Service No. ED464608) [verified 15 Apr 2007] http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ ERICServlet?accno=ED464608 Gosper, Woo, Muir, Dudley and Nakazawa 247 Gengoro. http://www.gengoro.com.au/ [viewed 7 Aug 2006] Gillani, B. (2003). Theories and the design of e-learning environments. Lanham, Maryland: University Press of America. iLecture – Lectopia. http://ilectures.uwa.edu.au/ [viewed 7 Aug 2006] Kulik, C. C. & Kulik, J. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behaviour, 7, 75-94. LAMS. http://www.lamsinternational.com/ [viewed 7 Aug 2006] McKeague, C. A. & Di Vesta, F. J. (1996). Strategy orientations, learner control and learning outcomes: Implications for instructional support of learning. Educational Technology Research and Development, 44(2), 29-42. Macdonald, J. (2002). Developing competent e-learners: The role of assessment. Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria , 28-30 August 2002 [viewed 7 Aug 2006] http://www.leeds.ac.uk/educol/documents/00002251.htm Mioduser, D. & Nachmias, R. (2002). WWW in education: An overview. In H. Adelsberger, B. Collis & M. Pawlowsky (Eds.), Handbook on information technologies for education & training. Berlin/Heidelberg/New York: Springer. Oblinger, D. & J. Oblinger (2005). Is it age or IT: First steps toward understanding the net generation. In D. Oblinger and J. Oblinger (Eds), Educating the net generation. Educause. [viewed 7 Aug 2006] http://www.educause.edu/educatingthenetgen Phillips, R. & Lowe, K. (2003). Issues associated with the equivalence of traditional and online assessment. In G. Crisp, D. Thiele, I. Scholten, S. Barker and J. Baron (Eds), Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference. Adelaide, 7-10 Dec 2003. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/419.pdf Reeves, T. (1997). Evaluating what really matters in computer-based education. [viewed 31 May 2006] http://educationau.edu.au/jahia/Jahia/pid/179 [for figures, see http://www.educationau.edu.au/jahia/webdav/site/myjahiasite/users/root/public/] Tergan, S. (1998). Checklists for the evaluation of educational software: Critical review and prospects. Innovations in Education and Training International, 35(1), 9- 20. Wimba. http://www.horizonwimba.com/ [viewed 7 Aug 2006] WebCT. http://www.webct.com/ [viewed 7 Aug 2006] Dr Maree Gosper, Karen Woo, Institute of Higher Education Research and Development, Macquarie University, NSW 2109 Australia. Email: maree.gosper@mq.edu.au, karen.woo@mq.edu.au Helen Muir, Christine Dudley, Kayo Nakazawa, Department of Asian Languages, Division of Humanities, Macquarie University, NSW 2109 Australia. Email: helen.muir@mq.edu.au, christine.dudley@mq.edu.au, kayo.nakazawa@mq.edu.au