mcalpine Australian Journal of Educational Technology 2001, 17(2), 115-130 Problem based learning in the design of a multimedia project Iain McAlpine The University of New South Wales Rex Clements The University of Melbourne This article examines problem based learning (PBL) as a basis for the design of interactive multimedia. To be effective, interactive multimedia should be stimulating and challenging, while providing students with a learning environment in which they can carry out investigations and have access to resources and tools for finding and manipulating data. PBL is an ideal approach to instructional design for multimedia, as it provides a scenario of problem presentation, investigation, resolution and presentation that can form the basis for the multimedia activities. Carried out effectively, this approach can enable high level learning, as the students are required to use the higher order mental processes of analysis, comparison and contrast, hypothesis and synthesis in order to propose a solution to the problem. Students need to take an active rather than a passive approach to learning when using this method. The theoretical basis for this approach is discussed. A case study of higher education course materials in grazing management is used to illustrate the way the PBL approach is used in the instructional design of the materials. Evaluation data from trials with students is included to illustrate the effectiveness of the materials for teaching and learning. Introduction An important part of the rationale for using multimedia and online technologies is to bring about an enhancement of learning outcomes. A recent review of student learning found deficiencies in relating basic science to applied applications, limited communication, problem solving skills and social aspects of the professional area, and an excessive volume of content (Keppell, Elliott, & Harris, 1998). These shortcomings limit their ability to function in the real world environment they will experience on graduation. Problem based learning (PBL) is an approach to teaching and 116 Australian Journal of Educational Technology, 2001, 17(2) learning developed to enhance the student’s ability to function in a professional environment. Its function is to make the process of solving high level complex tasks the major focus of the students’ learning. Students are required to actively engage in problem solving in a way that develops higher level cognitive processes such as investigation and analysis. This has the effect of building higher levels of skill. Using PBL as the basis for multimedia design, carried out effectively, will enable the high level learning outcomes associated with PBL to be attained by students who use the multimedia package. Multimedia technologies can be used to facilitate many of the learning processes that students need to engage in while learning in a PBL environment (McAlpine & Jackson, 2000). This suggests that PBL is a good teaching and Learning model on which to base multimedia design, as it will lead to higher level learning outcomes. It also means that teachers who wish to apply PBL in their classrooms will gain valuable assistance from using multimedia technology. PBL and approaches to teaching and learning Problem based learning was first developed in response to apparent shortcomings in students’ abilities to function in a complex professional environment. Some of the difficulties associated with traditional curricula in this area were: the curriculum was directed towards learning facts and technical skills without concern for understanding and reasoning; examination systems promoted rote learning rather than understanding; students were required to learn excessive and irrelevant detail; and it was doubtful whether students had a broad enough understanding and experience to carry out independent practice (Shanley & Kelly, 1994, p. 1). These issues are not limited to medical faculties. Koschmann et al., (1996, p. 85) conclude from studies of the literature that: … existing educational systems are producing individuals who fail to develop a valid, robust knowledge base; who have difficulty reasoning with and applying knowledge; and who lack the ability to reflect upon their performance and continue the process of learning. To overcome these difficulties a student centered multidisciplinary approach was developed at McMaster University in Canada, and a ‘hybrid model’ using problem based tutorials, lectures, conferences and clinical sessions around weekly themes was developed at Harvard University. These approaches enabled the students to be more engaged in independent study by using small group problem solving sessions that integrated learning across disciplines (Boud & Feletti, 1997). The major McAlpine and Clements 117 emphasis of PBL is to ground the student’s learning in the kind of situation they will face as a working professional. Students are presented with scenarios in which ‘real’ problems need to be solved. These are used as a basis for learning activities that are sufficiently comprehensive for the student to be able to deal with the full complexity of the situation. Types of problems A desirable attribute of problems used in PBL is that they should be ‘ill- structured’ (Koschmann, et al, 1996). Many academic tasks require well- structured problem solving, in which the student works towards a convergent or single known solution. Many of the situations that students will experience when they graduate however, are complex and are open to several different solutions. Dealing with problems of this nature may require different skills from problems that have a single solution (Jonassen, 1997). Problems are ill-structured or ill-defined if the problem statement does not readily specify the solution so that the learner must learn to identify the real nature of the problem, any information that is necessary but missing from the problem statement, and a process for arriving at a solution (Jonassen, 1997). Complex problems take many forms, depending upon the discipline area. Keppell, Elliott, & Harris (1998) describe a problem of the week approach in a medical faculty. Each problem has an audiovisual 'trigger' with supporting information in the form of a history, past medical history, physical examination, investigation results, and self assessment items. Reeves and Laffy (1999, p. 221) describe an example from an engineering course, in which students are given three major tasks: “1, transporting a research team to Mars; 2, siting and constructing a research station on the Martian surface; and 3, operating and maintaining an energy plant there”. As there are no clearly defined solutions to these problems, students must work out their own interpretation of the problem, then design and test models to reach their conclusion. They are then required to report their solution to the whole class and the teachers. Teaching, learning and assessment The most important value of PBL in teaching and learning is that the student needs to respond actively to the learning task. In this active response, the student must focus on the thought processes required to solve the particular problem. In cognitive terms, this process is at a much higher level than acquiring information in the form of facts, formulas or procedures. Rather, the student needs to learn the necessary facts, 118 Australian Journal of Educational Technology, 2001, 17(2) formulas or procedures in the context of using them to function at higher cognitive levels of analysis and synthesis to reach a solution that can be applied in the realistic situation of the problem scenario. The processes involved in PBL are important to the student’s academic development. By dealing with real scenarios the student is exposed to the reality that problems are not always clearly defined with single algorithmic solutions. In a PBL environment the student learns to deal with complexity and ambiguity by refining, often in collaboration with other students, just what the parameters of the problem are and how to go about reaching a solution (de Grave, Boshuizen, & Schmidt, 1996). Collaborative work can be an important feature of PBL environments (Oliver & Omari, 1999). Much of the development of PBL has been in a tutorial environment in which students share ideas and work together to solve the problem (Boud & Feletti, 1997). By working in groups the students develop skills in communication and teamwork. By negotiating a shared understanding, students learn to accept the points of view that other people bring to the problem. This creates the potential to achieve a deeper understanding of the subject (Jonassen, 1997). The nature of the assessment task is important. The PBL approach revolves around problem solving from real problems. The students, therefore, need to be assessed on their solutions to these problems. An approach to assessment that requires memorisation of large amounts of subject content, such as an examination, is inappropriate in a PBL environment as this approach does not necessarily assess the higher order analytical and problem solving processes that the PBL approach aims to develop. Assessment tasks in PBL should arise seamlessly from the learning activity in the same realistic or authentic manner as the problem is presented (Herrington & Herrington, 1998). Assessment must be based on the skills required in problem solving by assessing the performance of the skills in a realistic situation (Reeves & Okey, 1996). As small group work is often required, assessment may be on the basis of group assignments. This requires a process in which a group mark can be modified to take account of the relative performance of individuals. There are, however, acceptable methods of doing this in which the students provide feedback on relative performance of group members (Morris & Hayes, 1997; McAlpine, 2000). Basing assessment on the same activity as the learning task ensures that the intended skill development is assessed. McAlpine and Clements 119 PBL and approaches to learning The value of PBL as a way of encouraging higher level learning outcomes can best be described in relation to research on cognition and student learning. This research has identified the processes that lead to the deepest and most effective learning compared to the processes that are less effective. The former, or deep approach, involves actively seeking a meaningful interpretation of newly learned material to the extent that there is personal significance to the new knowledge and understanding that has been constructed. The latter, or surface approach, involved acquiring information for reproduction in an assessment task with little significance beyond this (Biggs, 1999). The difference between the two approaches is that the deep approach applies higher order learning processes such as comparing and contrasting, analysing, hypothesising and reflection, whereas the surface approach is oriented towards lower order processes such as describing, listing, or following simple procedures (Biggs, 1999). PBL is an approach to teaching and learning that requires the student to use higher order learning processes. Students who are presented with a complex problem need to undertake higher order processes such as analysing, comparing and contrasting, explaining causes, and hypothesising, and to apply the outcome of these processes towards the development of a solution. A student who does not apply these processes simply cannot complete the task as lower order processes such as memorising, describing, or following a simple procedure are insufficient. Group work and collaborative tasks lead to deeper learning as the cognitive processes of knowledge activation and elaboration, restructuring of existing and newly acquired knowledge are intensified through the questioning and evaluation processes necessary for group work (de Grave, Boshuizen, & Schmidt, 1996). As the problems used in PBL are typically complex, students are required to use ‘deep’ processes such as attending to contradictory information, attempting to understand alternative theories or explanations, and considering a wide range of evidence (de Grave, Boshuizen, & Schmidt, 1996). They need to find ways of reinterpreting knowledge and data to reach a solution. Biggs (1999) argues for an ‘aligned’ system of instruction in which objectives state the level of understanding required rather than listing topics, the teaching methods are selected to realise the objectives, students do the tasks nominated in the objectives, and the assessment tasks address the objectives. He considers that PBL is a perfect example of alignment as the: “objectives are to get the students to solve problems they will meet in 120 Australian Journal of Educational Technology, 2001, 17(2) their professional careers; the teaching method is to present them with problems to solve; the assessment is based on how well they solve them.” (Biggs, 1999, p. 71). The goal he proposes is for teachers to arrange the student activities so that the students are required to use higher order learning processes. Doing this requires a deep approach. PBL is an approach to teaching and learning that achieves these aims. Summary PBL is a powerful approach to teaching and learning as it requires the students to apply higher order learning processes that lead to deeper learning outcomes. It also focuses student learning towards the type of situation they will be working in as a graduate, and builds confidence in being able to function in these roles. By using this approach to design instructional multimedia materials, the power of PBL can be applied, and enhanced, using instructional technologies. Instructional design for multimedia and online technologies In a situation in which technology is being used to support the implementation of PBL the challenge for instructional designers is to create instructional materials that facilitate problem presentation and the required investigation, analysis and interaction. This includes group work on the problem where required and, possibly, the final presentation of a solution. The design team needs to recognise the cognitive processes required, and to facilitate these processes. The value of PBL is that it requires students to actively use the higher order cognitive processes that lead to deeper learning (Biggs, 1999). The instructional technology needs to be designed around these processes to ensure that they occur. These cognitive processes can be categorised as: • attending to and perceiving the nature of the problem and relating this to prior experience, • investigation and analysis of the problem and the resources available to attempt a resolution, • comparing and contrasting, hypothesising and synthesising in order to reach a solution, and • expressing the solution in a final form (Hoffman & Ritchie, 1997; Biggs, 1999). McAlpine and Clements 121 Appropriate instructional design and development of resources are needed to apply instructional technology to enable these learning processes. Each of the above processes can be supported by instructional technology as follows. Attending to the problem Given the nature of PBL, the learner needs a representation of the problem that is as close as possible to the real task (Hedberg, Brown, & Arrighi, 1997). By use of graphics, animation, audio or video, the learner can experience an information-rich representation that can alert the learner to important physical and social contexts that are relevant to the task (Hoffman & Ritchie, 1997). This leads to greater cognitive fidelity between the real world and the learning task. Investigation of the problem Students need a range of sources of information in order to investigate the issues raised by the problem presentation. In many cases the problem statement does not specify the information needed for a solution. Students need to be able to investigate a range of sources of information to enable them to understand the broader context in which the problem exists, to find additional information that may be needed to broaden their conceptualisation of the problem so that they can begin to hypothesise a solution. Instructional technology can be used to gather, annotate, and link a range of resource materials so that the students can exercise choices about what they select and the sequence as they carry out their investigation. This can be done with great efficiency as one source can link to everything and students can select materials at the time that they need them (Hoffman & Ritchie, 1997). Students interact with the materials in the genuine sense of interactivity. This is not merely selecting among options, but thinking about the problem before making the next investigation needed for a solution (Hedberg, Brown, & Arrighi, 1997). Development of a solution Resources for problem solving may include databases, spreadsheets, simulators, and other forms of software that are used as ‘cognitive tools’ in analysis and problem solving (Jonassen & Reeves, 1996). These ‘tools’ can enable learners to choose their own representation of a problem (such as selecting their own data to graph for comparison rather than accepting those of others) and promote deep reflective thinking. In this way, learning can be a challenging process that actively engages the learner 122 Australian Journal of Educational Technology, 2001, 17(2) (Reeves, Laffey, & Marlino, 1997). The key to the design of the materials is to ensure that the appropriate tools, such as comprehensive and accurate databases or pre-designed spreadsheets, are built into the materials with clear guidelines as to how to use them. PBL is usually structured around group work, to encourage additional thinking and sharing of ideas relating to the problem. By taking account of the views of others, students must reflect on their own ideas, challenge and expand their own conceptions, and engage in further learning to reach a solution. This can be carried out in face to face tutorial groups, or it can be achieved using synchronous or asynchronous computer conferencing (McAlpine, 2000). Presentation of a solution To maintain authenticity to the task, students should present a solution using the approach that is likely in the type of scenario presented in the problem statement (Hoffman & Ritchie, 1997). Instructional technology can be used as the tool for presentation if this is appropriate to the real situation, such as making a computer based slide presentation including graphs from a spreadsheet analysis (Jonassen & Reeves, 1996). Appropriate presentations may also take the form of a written report. An integral part of using instructional technology for PBL is the final delivery of a solution in whatever format is required. Summary Instructional design for PBL essentially involves creating the computer based framework, presentation mechanisms (such as audiovisual), access to the range of learning resources (including the creation of these resources), and creating or facilitating the tools needed for analysis and presentation. A case study using PBL for multimedia: Grazing management This case study considers a multimedia project at the Institute of Land and Food Resources, involving the teaching of grazing management practices. The subject requires the students to have a comprehensive practical understanding of how to manage pasture. A particular need is that students on several campuses, and possibly external students, need to study the same subject. A multimedia approach, using PBL, was seen to be the solution. This approach can enable all students in the subject to access the same resources, to engage in the same problem solving tasks, and to McAlpine and Clements 123 have access to specialist computer based tools such as specialised spreadsheets and a simulator, to aid them in reaching appropriate solutions to the problems. The students can deal with authentic, complex problems without the need for field trips to different regions in the state to experience the problems associated with pasture management in different climate zones. The multimedia materials need to enable the students to: understand the major influences on pasture growth; assess the merits of different pasture species and cultivars; understand different pasture growth systems and the effect of grazing animals; and be able to budget and manage pasture on a systematic basis. The particular attributes the multimedia materials needed to have to apply a PBL approach were: • a realistic presentation of genuine problem scenarios; • access to a range of resources (information and tools) for problem solving; • facilitation for individual and group work; • facilitation for realistic problem presentation; • authentic assessment based on problem solving. Instructional design for the multimedia materials focused on creating materials that would facilitate or promote the learning processes related to PBL, using the above key attributes. These were implemented in the design in the following way. • Problem scenarios are presented primarily in audiovisual format. In the first module this consists of: photographs of the farmers and of the farm itself showing wide views of the overall farm layout; closer views of the paddock in need of renovation on each farm; and close up images of the pasture illustrating problems such as the predominance of weed species or bare patches; a recorded interview with the farmer enabling the student to learn about the farm, the climate, the intended use of the pasture, and the farmer’s perception of the problem. These are presented to the student in a slide show format. • Students are guided to solve the problem using a range of resources and tools. The resources include computer based tutorials that explain and illustrate important concepts such as the influence of rainfall and soils on pasture, and maps illustrating pasture or climate zones. The tools include searchable databases such as a pasture species database, and a pasture growth simulator developed as part of a research project. 124 Australian Journal of Educational Technology, 2001, 17(2) Students are able to use any or all of these resources to work out their solution to the problem. • The instructional materials are designed to be usable individually or for groups. The problem scenarios are sufficiently complex to be suitable for group work. Individual lecturers using the materials can decide whether there will be tutorial groups facilitated by the lecturer, or student groups, or individual activity. By using computer conferencing external students can participate in collaborative group activity. • Students need to present their problem solution in a realistic way. Students are provided with a template for a report format, and are able to copy and paste data from their analysis, such as a spreadsheet output, or data from the simulation. Students may use computer generated slides to make individual or group presentations, also using data that is sourced from the problem solving tools. • The students’ assessment will be based on problem solving ability, including evidence that the student can use data analytically, rather than simply arriving at a solution by guesswork. The solutions that the students present will be in a format that is realistic to the problem task, such as a recommendation to a farmer or a presentation to a cooperative board. Assessment tasks are authentic as they closely relate to the authenticity of the original scenario. Evaluation Formative evaluations were carried out with two small groups of students, using a regular class time of four hours in the computer laboratory, so that all students had access to a networked computer. The format for the classroom sessions was: • 30 minutes introduction to the topic by the subject lecturer; • 2 hours of individual study time using the project materials; • 1 hour for group discussion of the problem and group presentations of a solution, and • 30 minutes to complete questionnaires and a focus discussion. The study considers four specific indicators of the students’ progress and the effectiveness of the role played by the computer based PBL materials: 1, observation during the class session; 2, questionnaires with Likert scale McAlpine and Clements 125 questions and space for open ended comments completed by the students at the end of the session; 3, focus discussions between the authors and students at the end of the session; and 4, observations by the staff relating to the quality of the students’ output of solutions to the task. Observation during class session In the first trial the materials were slow to download which caused some frustration, however the students were sufficiently interested in the task to persist. An important part of the learning resources for the task is a pasture species database that provides a wide range of information on each plant species and cultivar. The students appeared to find this very valuable and absorbing. The students used this frequently during the preparation of both their individual and group solutions to the problem. Questionnaires The questions are organised into four topic areas: 1, questions on the use of technology; 2, questions relating to cognition, including aspects such as activation of prior knowledge, level of challenge, and development of new knowledge and skill; 3, effectiveness of group work in developing a deeper level of understanding; and 4, the students’ subjective view of their own learning. Table 1: Questions relating to the use of technology SA - strongly agree à SD - strongly disagree SA A NS D SD 1 The way to use the online technologies was made clear to me from the beginning. Gp 1 3 3 3 Gp 2 2 5 2 I experienced difficulties in gaining access to the online subject materials. Gp 1 2 3 4 Gp 2 1 2 3 1 3 After the initial problems of connection were overcome, access to the online materials was consistent and effective. Gp 1 7 2 Gp 2 3 3 1 4 The multimedia based technology helped me to learn effectively. Gp 1 2 5 1 1 Gp 2 2 3 2 Table 1 illustrates the questions and the levels of student response that relate to the technology itself, for each of the two groups of students who participated in the evaluation. These indicate that while there were some difficulties with access to the materials, these were overcome and the materials were perceived to facilitate effective learning. 126 Australian Journal of Educational Technology, 2001, 17(2) Table 2: Questions on cognition and learning SA - strongly agree à SD - strongly disagree SA A NS D SD 5 The best way to approach learning was clearly explained in the online subject materials. Gp 1 1 3 1 4 Gp 2 1 5 1 6 The problem situation had some features that were familiar to me. Gp 1 3 5 1 Gp 2 1 5 1 7 I found the problem situations challenging. Gp 1 2 4 1 2 Gp 2 5 2 8 The problem tasks enabled me to develop my existing knowledge of the topic. Gp 1 2 7 Gp 2 1 5 1 9 I gained a greater understanding of how to meet farmers' needs from working on the problem solving tasks. Gp 1 2 7 Gp 2 1 5 1 Table 2 illustrates the questions and levels of response on some issues relating to cognition. These deal with orientation, activation of prior knowledge, development of existing schemata, and the level of outcome in terms of skill development. The trend is for students to feel that they developed their knowledge and skill as a result of the activity. The open ended comments following this group of questions were generally favorable about the subject material. These are exemplified by this comment: “Exactly what a farmer would want when selecting pasture species.” Table 3: Questions on group work SA - strongly agree à SD - strongly disagree SA A NS D SD 10 I had to consider several different points of view in the group discussion of the problem tasks. Gp 1 1 8 Gp 2 2 4 1 11 I worked closely with other students on the group learning tasks. Gp 1 1 7 1 Gp 2 2 4 1 12 I learned a lot from the other students while working on the group learning tasks. Gp 1 8 1 Gp 2 1 2 3 1 Table 3 illustrates the questions that concentrate on the value of group work in PBL. The responses to these questions indicate that the students placed a high value on the group activity. The open ended responses after this set of questions support this observation, as illustrated by the McAlpine and Clements 127 following comment: “It helped to make sure I could justify the choices I made and did not allow you to go through and just pick cultivars”. Table 4. Questions relating to overall level of learning SA - strongly agree à SD - strongly disagree SA A NS D SD 13 The subject lecturer provided effective guidance and feedback. Gp 1 1 8 Gp 2 1 5 1 14 I felt that I developed a deep understanding of the subject content. Gp 1 7 1 1 Gp 2 4 2 1 15 My studies in this subject helped me to develop problem solving skills that will be useful to me professionally. Gp 1 2 5 2 Gp 2 1 5 1 16 I learned a method of approaching new problems by carrying out the problem tasks. Gp 1 2 5 2 Gp 2 1 5 1 The questions in Table 4 relate to perceived level of understanding and skill development in areas that the students will need to practice following graduation. The responses are generally favourable in this area also, indicating a perception of skill development that will be of value to the student as a working professional. This perceived value is illustrated by this student comment: “I think that it is great being able to relate to situations right back to the paddock”. Lecturer assessment of the quality of outcomes In comparison to the equivalent group of students from the previous year, the following general observations were made: 1. Got the students interested and working for 4 hours! Though slow to start, once into the program, the students did get interested to find out more and also complete the task in a methodical. The progression of the subject material in the module, definitely works! The problem based approach sustains interest until the final solution is arrived at. 2. Satisfactory results from the whole group. The concepts were driven in a more effective manner than in the classroom. Being non-technical but with a logical progression, the students were able to absorb the material at their own pace – some quicker than others. The group discussion assisted in the transfer of knowledge and experiences - also kept them more focused. This definitely helped the bottom third of the 128 Australian Journal of Educational Technology, 2001, 17(2) class, to catch up on the others – with every student being involved in the learning and assessment at the same time. The minimum expectations of the whole group were clearly achieved. 3. Useful tool to supplement lectures/practicals. Field visits/practical classes will be more meaningful for students, to verify the concepts learnt in this package. The module is an effective means of introducing the theoretical concepts and stimulating the thinking process of the students. It could be effectively used in the delivery of this subject by distance education. Conclusions PBL appears to have a high potential as a basis for the design of multimedia materials in this subject area. Instructional design for multimedia and online technologies using a PBL approach must be concentrated on using the technologies to enable and support all of the essential processes required for this approach to teaching and learning. All of the normal processes of instructional design relating to needs and task analysis, setting objectives, content analysis and conceptual structure, cognitive mapping of conceptual relationships are still applicable when developing for a PBL approach. It is essential to have a thorough grounding for the content material. When this level of analysis is complete, however, design for PBL involves the provision of resources for problem solving. Content materials may be structured so that they give the appearance of the materials that are used in real situations, such as lab reports or records of applications. Materials must be designed to provide appropriate access to these resources. The other major aspect of design is to provide effective tools for problem solving. Templates, spreadsheets, graphing tools, simulators provide a focus for learning activities. Materials need to be designed to enable students to work individually or in groups, perhaps using online conferencing. In these ways, the multimedia materials can be structured to support the appropriate learning processes. In this case, the multimedia materials were effective in promoting the required learning outcomes. After graduation many of these students will work in a consultancy role in which problem solving is a critical skill. The multimedia format enabled the students to experience many of the issues associated with a real problem, and to quickly access the resources to enable a solution to be reached. The subject lecturer feedback indicated that the quality of learning was high. Student feedback indicated that they perceived the value of the activity in building skills in such a role. By extrapolation, this approach to using PBL for multimedia design can be McAlpine and Clements 129 applied to many other subject areas also. PBL is applicable in any subject area where investigation is needed because solutions are not algorithmic processes with specific answers. As this includes a wide range of subject areas, this approach to instructional design has the potential for a wide range of applications. References Biggs, J. (1999). What the student does: Teaching for enhanced learning. Higher Education Research and Development, 18(1), 57-75. Boud, D. & Feletti, G. I. (1997). The Challenge of Problem-Based Learning, 2nd Ed, London: Kogan Page. de Grave, W. S., Boshuizen, H. P. A., & Schmidt, H. G. (1996). Problem based learning: Cognitive and metacognitive processes during problem analysis. Instructional Science, 24, 321-341. Hedberg, J., Brown, C., & Arrighi, M., (1997). Interactive multimedia and web- based learning: Similarities and differences. In B. H. Khan (Ed), Web-Based Instruction. Englewood Cliffs NJ: Educational Technology Publications. Herrington, J., & Herrington, A. (1998). Authentic assessment and multimedia: How university students respond to a model of authentic assessment. Higher Education Research and Development, 17(3), 305-322. Hoffmann, B., & Ritchie, D. (1997). Using multimedia to overcome the problems with problem-based learning. Instructional Science, 25, 97-115. Jonassen, D. H. & Reeves, T. C. (1996) Learning with technology: Using computers as cognitive tools. In D. H. Jonassen (Ed), Handbook of Research on Educational Communications and Technology. New York: Simon & Schuster Macmillan: 693- 719. Jonassen, D. H. (1997). Instructional design models for well-structured and ill- structured problem solving learning outcomes. Educational Technology Research and Development, 45(1), 65-94. Keppell, M., Elliott, K., & Harris, P. (1998). Problem based learning and multimedia: Innovation for improved learning of medical concepts. Proceedings ASCILITE ’98, The University of Wollongong, 14-16 December. [verified Sept 2000] http://www.ascilite.org.au/conferences/wollongong98/ascpapers98.html Koschmann, T., Kelson, A. C., Feltovich, P. J. & Barrows, H. S. (1996). Computer- supported problem-based learning: A principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed), CSCL: Theory and Practice of an Emerging Paradigm. Mahwah, NJ: Lawrence Erlbaum. 130 Australian Journal of Educational Technology, 2001, 17(2) McAlpine, I. (2000). Collaborative learning online. Distance Education, 21(1), 66-80. McAlpine, I. & Jackson, J. (2000). Implementing Online Delivery: A Study of Change Management. Fellowship Report for the Australian National Training Authority (ANTA). Brisbane: ANTA. http://www.anta.gov.au/fellow/ Morris, R. & Hayes, C. (1997). Small group work: Are group assignments a legitimate form of assessment? In Pospisil, R. and Wilcoxson, L. (Eds), Learning Through Teaching, p229-233. Proceedings of the 6th Annual Teaching Learning Forum, Murdoch University, February 1997. Perth: Murdoch University. http://ctl.curtin.edu.au/events/conferences/tlf/tlf1997/morris.html Oliver, R. & Omari, A. (1999). Using online technologies to support problem based learning: Learners' responses and perceptions. Australian Journal of Educational Technology, 15(1), 58-79. http://www.ascilite.org.au/ajet/ajet15/oliver.html Reeves, T. C. & Okey, J. R. (1996). Alternative Assessment for Constructivist Learning Environments. In B. G. Wilson (Ed), Constructivist Learning Environments, Englewood Cliffs, NJ: Educational Technology Publications. Reeves, T. C., Laffey, J. M., & Marlino, M. R. (1997). Using technology as cognitive tools: Research and praxis. Proceedings ASCILITE ‘97, Curtin University, December 7-10. [verified 26 Jan 1999] http://www.curtin.edu.au/conference/ASCILITE97/papers/Reeves/Reeves.html Reeves, T. C. & Laffey, J. M. (1999). Design, assessment, and evaluation of a problem-based learning environment in undergraduate engineering. Higher Education Research and Development, 18(2), 219-232. Shanley, D. B., & Kelly, M. (1994). Why Problem-Based Learning? http://www.odont.lu.se/projects/ADEE/shanley.html [verified 19 Aug 1998]. Dr Iain McAlpine Flexible Education Developer Educational Development and Technology Centre (EDTeC) The University of New South Wales, Sydney 2052, Australia Email: i.mcalpine@unsw.edu.au http://www.edtec.unsw.edu.au/ Dr Rex H G Clements Animal Production Institute of Land and Food Resources, Glenormiston The University of Melbourne rclements@unimelb.edu.au http://www.landfood.unimelb.edu.au/