liaw-ed Australian Journal of Educational Technology 2000, 16(2), 104-125 Using multimedia to assist students with communication skills and biopsychosocial integration: An evaluation Teng Liaw, Gregor Kennedy, Mike Keppell, John Marty and Ruth McNair Faculty of Medicine and Dentistry and Health Sciences The University of Melbourne This paper reports on the development rationale and evaluation of a computer facilitated learning (CFL) package which aimed to assist medical students with their clinical communication skills and develop an integrated biopsychosocial approach to identifying a patient’s problems. Using a formative evaluation framework developed at the University of Melbourne the CFL package, Communicating with the Tired Patient, underwent a three stage review. Initial evaluation consisted of both formal and informal conceptual and technical review by content experts, instructional designers and evaluators. The final stage of the evaluation involved the survey and observation of 110 medical students interacting with the package. Students were very positive about the instructional and interface design of the package and indicated that the package assisted with their understanding of issues associated with clinical communication. Nevertheless, a number of areas were highlighted where either the package or the learning environment could be modified or improved. Introduction Interactive educational multimedia packages, which combine robust learning design with innovative technical design, are regarded as an efficient and effective way to assist and foster students’ learning. Given this potential, there has been a concerted effort to produce educational multimedia that is both pedagogically and graphically sound. While a great deal of attention has been given to the process of producing educational multimedia, less attention has been given to the evaluation of this production. Typically formative evaluation has been used to refine and improve educational multimedia while it is under development. There are a number of advantages to having a clear program of formative evaluation. These include (i) saving time and money in development and Liaw, Kennedy, Keppell, Marty and McNair 105 redevelopment (ii) maximising the potential for developing educationally and graphically sound multimedia and (iii) providing an evaluation structure that leads logically into summative, impact or integrative evaluation. This paper describes the learning goals, educational context and learning design of a specific educational multimedia package, Communicating with the Tired Patient, and reports on its formative evaluation. Learning goals of the package Identifying a patient’s problems in a clinical interview is a complex skill that health professional students must acquire. Too frequently health professional students fix on the biological aspects of a patient and ignore social and psychological factors that may be contributing to the patient’s problems. Communicating with the Tired Patient, aimed to alert students to the need to consider all facets of the patient's history in an integrated biopsychosocial approach. A second aspect of a clinical interview, or any interpersonal interaction, is the communication microskills used by the participants. Microskills refer to the interviewing strategies that are used to help facilitate a successful interview and to establish rapport between the interviewer and the interviewee. Microskills may be verbal or non-verbal. Verbal strategies include using different types of questions (open, closed or leading), reflecting, paraphrasing, minimal encouragers and using different tones of voice. Non-verbal strategies include posture and facial expressions, and encouragers like nods and eye contact. A skilled clinician listens to verbal responses and observes the non-verbal behaviour of the patient and is able to monitor how comfortable and forthcoming a patient is and alter his or her interview accordingly. Thus, the computer facilitated learning (CFL) package Communicating with the Tired Patient had two overarching aims. First, it aimed to help students develop an understanding of the need to consider biological, psychological and social factors when conducting a clinical interview and ultimately in their identification of a patient’s problems. Second, the package challenged students to reflect on an array of microskills used and a range of verbal and non-verbal behaviours exhibited, by both the doctor and the patient, in a clinical interview. Context of use Communicating with the Tired Patient was initially designed to be used with first year medical students at the University of Melbourne. In 1999, the 106 Australian Journal of Educational Technology, 2000, 16(2) University of Melbourne introduced a new medical curriculum incorporating problem-based learning, self-directed learning and educational technology (Keppell, Elliott & Harris, 1998). In the new curriculum, students are presented with a weekly clinical problem which they are required to investigate using self-directed learning resources, including text books, journal articles, web sites and CFL modules, which are provided for them. The integration of biomedical and clinical components of the course and a greater emphasis on self directed learning indicated a need for resources that directly addressed clinical skills. Thus, Communicating with the Tired Patient was designed to be used as a stand- alone CFL module which students could use as a self-directed learning resource. The package was made available to students in a computer laboratory which housed fifty desktop computers and was also located on computers in students’ problem based learning tutorial rooms. Each computer was equipped with headphones to allow students uninterrupted use of the package. Learning design and description of the package Communicating with the Tired Patient was designed so that the student user was the focus of the package and was an active participant in an interactive, virtual interview. Other packages have tackled the issue of communication skills by simply presenting various interview situations to students and asking them to reflect on them (Medical AudioVisual Communications Inc, 1998). The developers of the current package were keen to move way from this didactic, “fly on the wall” approach, by enabling students to make decisions about how they wanted their interview to be conducted. In the package students are asked to play the role of the doctor in a clinical interview. As the doctor students are given a number of options regarding the types of questions they are able to ask the patient which reflect different microskills common in clinical interviews. Students are asked to listen to the different questions and to select one they would like to ask the patient. Students are able to see the ramifications and implications of asking different questions by listening to and observing the patient's video response. Consistent with the concept of a virtual interview, the patient responds directly to the student, as in Figure 1. Liaw, Kennedy, Keppell, Marty and McNair 107 Fi gu re 1 : A s cr ee n fr om th e pa ck ag e sh ow in g th e ro llo ve r au di o, th e vi de o of th e pa ti en t a nd th e fe ed ba ck a nd q ue st io ns u se d to p ro m pt s tu de nt s. 108 Australian Journal of Educational Technology, 2000, 16(2) To assist them with their reflection, questions which are specific to the last audio-video segment pop-up on screen which students can answer in a workbook which supplements the package. Students are encouraged to comment on what they noticed about the patient's verbal and non-verbal responses and are challenged to think about the direction in which their interview is heading. In addition to considering the microskills which are reflected in students’ choices, students are asked to reflect on the possible causes of the patient’s presenting complaint. This process aims to promote students’ understanding of the biopsychosocial aspects of a patient’s presenting problems. The development team saw a number of advantages in developing the package in this way. First, students not only actively participate in the interview with the patient but they actively construct it as well, controlling both its tenor and direction. Each student is able to select the questions, options and pathways in a clinical interview which make sense to him or her. By interacting with the package in this way, it is hoped that students will build up new understandings in the content area based on their current—and often implicit—knowledge of appropriate interviewing methods. Kennedy and McNaught (1997) suggest that one of the critical challenges facing educators is “enabling passive learners to use more active approaches to learning and new knowledge construction...” (p. 391). Communicating with the Tired Patient aims to confront this challenge head on by actively involving and engaging students in the learning experience rather than simply presenting information for them to absorb. The theoretical underpinnings of the package are, therefore, consistent with a constructivist philosophy of courseware design and development. A second important aspect of the learning design of the module was the promotion of critical reflection by students. Boud, Keogh and Walker (1985) define reflection as “those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations” (p. 19). Schon (1983) describes reflection-in-action as analogous to “thinking on your feet” or “learning to adjust once you are out there” (pp. 54-55). It is reflecting in the midst of an operation rather than at a post-mortem. The structure of the package meant that students were involved in a process that required them to critically reflect on their decisions and tacit understandings during and after their virtual interview. The package helped students, both implicitly and explicitly, to reflect on their choices. After students have selected a Liaw, Kennedy, Keppell, Marty and McNair 109 particular doctor’s question they are able to see the implications of their selection through the audio and video response of the patient (implicit feedback). In addition, students are supported with specific feedback and are challenged with further questions about their selections (explicit feedback). Through reflection students are able to critique how they have gone about their learning experience and identify areas where these processes could be improved (Koschmann, Kelson, Feltovich, & Barrows 1996; Schon 1983). By returning to their learning experiences, attending to their feelings and re-evaluating these experiences, students are able to associate and integrate information. This will hopefully foster greater understanding (Boud et al, 1985). Another advantage of the development team's approach was that through the use of audio and video and by placing the student in the role of the doctor, the developers attempted to create a realistic scenario. Using the model of situated cognition, the developers recognised that the media of audio and video could be used effectively to place learners in a real life context (Brown, Collins & Duguid, 1989; Herrington & Oliver, 1995, 1997). Brown, Collins and Duguid (1989) argue that “useful learning” takes place when students are set authentic tasks or are placed in authentic contexts. They argue that “much school work is inauthentic and thus not fully productive of useful learning” (p. 34). The challenge for multimedia developers, therefore, is to create learning activities and contexts which are authentic and maximise the potential for student learning and facilitate the application of what is learned. The developers attempted to meet this challenge with Communicating with the Tired Patient. A program of formative evaluation The Biomedical Multimedia Unit in the Faculty of Medicine, Dentistry and Health Sciences has developed an extensive program of formative evaluation to assist Faculty staff with the refinement and improvement of their educational multimedia packages while they are under development. The program of evaluation was established in order to maintain the quality of the increasing number of educational technology packages being developed in the Faculty. The overarching focus of the program is on the design of educational multimedia that will best support student learning. 110 Australian Journal of Educational Technology, 2000, 16(2) Details of this program are reported elsewhere (Kennedy, Petrovic & Keppell, 1998; Kennedy 1999a, 1999b), and will not be covered in detail here. However, the program has a number of key elements: • an emphasis on the iterative nature of formative evaluation which dovetails with an educational multimedia development cycle (Keppell & Bennett, 1997). This ensures that evaluation takes place from early stages in the development of CFL programs. • the need to consider three different evaluation domains and specific criteria within these domains. The evaluation domains include instructional and conceptual design, interface and graphic design and user attitudes and affect. • the need to consider a number of different evaluation perspectives including both internal and external reviews from subject matter experts, instructional designers, graphic designers, student users and educational evaluators. • the need to employ a variety of data collection techniques including questionnaires, checklists, focus groups, observations and expert reviews. This paper reports on the method of formative evaluation undertaken by the development team associated with Communicating with the Tired Patient, the results which emerged from this evaluation and the recommendations which ensued. Formative evaluation of Communicating with the Tired Patient The formative evaluation of Communicating with the Tired Patient which has been conducted to date, can be conveniently divided into three stages. In the first stage an initial conceptual and technical review was conducted by the development team, while in a pre-alpha version stage both informal and formal reviews were carried out of both instructional design and interface design. In the final stage a comprehensive formative evaluation of an alpha version of the package was carried out. This evaluation comprised both internal and external expert review, student user observations and the surveying of 110 medical students. Each of these stages will be reported on briefly below before more substantial results are presented for the final stage of this evaluation. Liaw, Kennedy, Keppell, Marty and McNair 111 Initial conceptual and technical review Formative evaluation during early stages of the package's development consisted of informal conceptual and technical review by content experts, instructional designers and evaluators. Story boards and concept maps were reviewed to determine the scope of the project, the feasibility of its completion given the resources that were available and the technical requirements of the proposed package. Through this process of informal review the development team was able to focus on the goals of the project, the learning design, how students would interact with the package and what, in fact, students would do when put in front of the product. The major concerns during this phase of the evaluation were reducing the size of the project so that it was manageable both conceptually and technically. A number of scripts were produced by the content experts for the proposed clinical interview case studies, showing the details of interactions between doctors and patients. There was a need to keep scripts as simple as possible without undermining the learning objectives. Scripts of relatively simple interactions were produced because of technical and storage limitations as well as the target audience of first year students. Once working versions of these scripts had been established, estimates were made of the file sizes associated with each of the clinical interviews. Given the extensive use of audio and video in the project it was decided that cross-platform CD-ROM was the most appropriate delivery mode. Due to the proposed number of clinical interview case studies, the length of these interviews and the large file sizes associated with capturing these interviews on audio and video, it was clear to the development team that compromises were needed in terms of production. These compromises were manifest in two ways. First, the number of clinical interview case studies that were initially proposed was reduced. Second, it was determined that video and audio would be produced for both the doctor and the patient for the reduced number of clinical interviews but the development team would initially focus on producing two of these as interactive learning modules. Pre-alpha version informal and formal review Once production had begun on the interface and graphic design of the package it was exposed to informal review by instructional designers, graphic designers and content experts. This informal review led to a number of modifications being made to the interface and graphic design of the package. The final aspect to this stage of the formative evaluation 112 Australian Journal of Educational Technology, 2000, 16(2) involved both an educational evaluator (GK) and an internal content expert (TL) formally evaluating a prototype of the package and preparing written reports. This review was completed with the aim of finalising any substantial alterations that were required before producing a working version of the package. This proved an extremely useful exercise. While a great deal of attention had been paid to the overall structure of the package and how students would interact with it, some of the micro details had been overlooked in production. Thus, the evaluation switched in focus from the macro structure, the complex programming and the technical requirements of the package to the micro details associated with student interactions. A number of major modifications were made to the package as a result of this evaluation. These modifications involved providing more support for students, both in terms of functionality and educational goals. In terms of functionality, the evaluator and content expert agreed that students may not be clear about how to negotiate their way around the package. The evaluation recommended more specific and clearer instructions be provided in the introduction. It was also recommended that students be provided with more specific on-screen instructions when selecting a doctor and a patient for the interview and when students were required to operate the audio functions. In terms of educational support, it was thought that the package was in danger of being too open and lacking an educational direction. There was a clear tension between adopting a constructivist development approach on the one hand and providing enough instructional support for students on the other. It was thought that if the package remained in its current form it would not be clear to students what the purpose of the package was. At this stage it was decided that students would be prompted to reflect on the specific audio-video interview segments they had chosen. The content expert provided an array of supporting questions to fulfil this function. For example, after a particular audio-video sequence a question would pop-up on the screen: “You asked Mrs Nacarella a closed question in a business like fashion. Was the response what you expected?” It was decided at this stage that a workbook containing all the questions available in the package would be used to supplement the package. In addition to these modifications a number of minor amendments were made to the package at this stage. A conclusion was included in the package to provide students with a summary of the biopsychosocial implications of the patient’s presenting complaint. A number of technical glitches, such as audio not being activated by rollovers and poor screen transitions, were noted and subsequently rectified. In one section (the Liaw, Kennedy, Keppell, Marty and McNair 113 glossary) the readability of the text was improved. Finally, areas of the package which were included in the initial interface design but currently lacked functionality (the tutorial and a number of clinical cases) were either removed from the interface or were clearly labelled as “under construction”. Internal and external formal evaluation of an alpha version of the product Once the alpha version was deemed ready, it was installed in the computer laboratory for use by first year medical students in the new problem based learning curriculum as a self directed learning package. One clinical interview case study was made available to students and was based on the consultation between a male doctor and a female patient from a non-English speaking background. This module was the focus of an extensive evaluation. Expert review of the module was provided by two internal and three external content experts. When students were using the package in the laboratory both general observations of student users and specific observations of three student users interacting with the package were conducted. Finally, 110 medical students were surveyed using a questionnaire containing both fixed response and open-ended questions. Although quantitative data was collected, generally there was an emphasis on qualitative data given that the aim of the evaluation was to refine and improve the package. Qualitative data (open ended responses and observations) are more useful for obtaining more specific details of how CFL packages can be altered and where possible improvements can be made. The data reported below represent the results from this final stage of the formative evaluation. Results A number of themes or areas were targeted in the alpha version evaluation. These ranged from a general evaluation of the package and the interface to a specific evaluation of the audio and video and how students used the workbook in conjunction with the package. In addition, a number of themes emerged from the open-ended questions that were asked of both students and content experts. The results section is, therefore, divided into a number of sections: audio and video, workbook, the use of questions, the package and interface generally, technical issues, and students understanding. 114 Australian Journal of Educational Technology, 2000, 16(2) Audio and video Students were asked to comment on the use of audio and video in an open text response. Students’ responses were coded thematically and frequency counts of the themes which emerged were recorded. Table 1 shows the eight highest ranked responses to the statement “Please comment on the use of audio and video in the program”. It was clear from students’ responses that they were positive about the use of audio and video, thinking that it improved the package. However, it was also clear that there were problems with the quality of the audio in some instances, with 29 students mentioning that the audio was too soft or unclear. This response was also reflected in a number of students commenting that the poor audio quality was one of the worst aspects of the package (21)1 (see Table 3 below). Table 1: Highest eight ranked responses to the statement “Please comment on the use of audio and video in the program”. Response Frequency The video was fine/good/improved the package 54 The audio was fine/ good/improved the package 49 The audio was too soft/unclear 29 The video was too small 9 The questions were too similar/repeated too often 8 You should be able to go back and repeat video showings 8 The interview was not sequential/in the wrong order 4 The video was blurry 3 Students were also asked to rate the quality of the audio and video on nine-point bipolar scales where ‘1’ indicated high quality and ‘9’ indicated low quality. Mean scores for the audio and the video indicated students thought that generally both were of high quality (audio mean = 4.08, SD = 2.34; video mean = 3.25, SD = 1.80). However, the high standard deviations, especially for the audio, indicated there was a high degree of variability in these responses. Paired T-tests revealed that students were more positive about the video than the audio (t (108) = 3.94; p < .001). 1 Numbers in parentheses represent frequency counts or percentages of a particular response made by students. This procedure is used throughout the results. Liaw, Kennedy, Keppell, Marty and McNair 115 Technical difficulties were experienced with volume control when the package was installed on computers in the computer lab. As such the audio problems experienced were not a result of the package per se, but resulted from its integration into the learning context. Students may have had difficulties hearing the audio for one of three reasons: their headphones were plugged into the wrong audio socket, the sound control software was set too low, or the mute button was on. Students were often not experienced enough with the computers to rectify this situation on their own. While content experts were not asked specifically about the audio and video, two mentioned its high quality. Content experts also noted the use of audio and video was excellent for creating a “real-life” scenario and for training students to listen to and observe their patients. Student observation supported this conclusion. Students spent a lot of time reviewing the audio options and some students closed their eyes when listening to the audio in order to focus on the difference between the audio options. One student was heard to say “No, no, no. I don’t want you to say that!” after making an audio choice that she was unhappy with. Students were also observed concentrating fiercely on the video responses of the patients in order to pick up both verbal and non-verbal cues. These observations suggest that the audio and video were useful in engaging students in the content of the package. This is reflected in the two most common comments by students regarding the best aspects of the package which were the video (46) and the audio (36) (see Table 5 below). It seems, therefore, that other than the technical difficulties surrounding sound quality, that the audio and video were well received. The workbook Students were also asked to comment on the use of the workbook in conjunction with the package. This was a point of interest for the development team as they wanted to ensure the workbook which was provided for students was used cohesively with the package and was a worthwhile component of their learning experience. Table 2 shows the ten most frequent responses to the statement “Please comment on the use of the workbook in conjunction with the program”. It is clear from these responses that students felt that the package and the workbook complemented each other and the workbook assisted them with their learning. A few content experts mentioned that the workbook was a useful tool for students and challenged them to make a response to the interview which was transpiring. This is clearly supported by students’ third most frequent response in the table below. 116 Australian Journal of Educational Technology, 2000, 16(2) Table 2: Highest ten ranked responses to the statement “Please comment on the use of the workbook in conjunction with the program”. Response Frequency The workbook and the program worked well together/they were easy to use together 42 Negative comments about the workbook questions 25 The workbook assisted learning/helped direct attention/was stimulating 18 Positive comments about the questions 9 The workbook interrupted the flow of the program 7 There was no need to write down so much in the workbook 6 It was difficult to know which question you were dealing with 5 It was difficult to switch between the workbook and the program 4 It was too confusing 3 The workbook left too much space for answers 3 Relatively few students thought the workbook affected the continuity of the program (7) or had trouble moving between the workbook and the program (4). When asked explicitly about moving between the workbook and the program the vast majority (72.7%) indicated they had found this easy with only around thirteen percent of students (12.7%) indicating this presented them with difficulties. Observations suggested that students were, on the whole, not having too many difficulties negotiating transitions between the package and the workbook. On occasions, students would be distracted from the screen when they looked for the next question in the workbook or reviewed their previous answers. The concern that the attention of some students may sometimes be divided between the workbook and the interview may be rectified by allowing students to "rewind" their interview. This is something the students themselves requested (see below). Students were, however, critical about the nature of the questions in the workbook. The most common criticism was that questions in the workbook were too repetitive or monotonous (12). This is consistent with students’ comments about the package structure generally (see Table 4 below) which is not surprising given students use of workbook is contingent on their path through the package. Other comments were that the questions were too difficult (7), were not in sequence (3) or were ambiguous (2). Liaw, Kennedy, Keppell, Marty and McNair 117 The use of questions Students were then asked about what they thought were the three worst aspects of the package. These general questions are useful in formative evaluation as they give users the chance to give their opinion of what specifically “didn’t work” in the package. This information is often very revealing and useful for developers, as it can indicate where problems occur and specifically suggest how the package can be modified. Table 3 presents the ten most frequent response to the statement “Please note what you thought were the three worst aspects of the package”. It is clear from students’ responses that they had difficulties with the Doctor’s questions in the virtual interview. The majority of the negative responses associated with the Doctor’s questions concerned the repetitious nature of these questions. It seems that students found themselves directed to the same part of the interview on a number of occasions and thus, they experienced the same Doctor’s questions on a number of occasions. This would also explain the comment made by students that the workbook questions were repetitious, as the workbook questions are contingent on where students go in the package. These comments are also consistent with students’ perceptions of the package generally: thirty-five percent of students found the package generally repetitive. Table 3: Highest ten ranked responses to the statement “Please note what you thought were the three worst aspects of the package”. Response Frequency The problematic nature of the Doctor’s questions 40 The options or paths were too similar or limited 24 The sound/volume was poor 21 There was no going back to repeat questions 16 The interview lacked continuity 12 It was too boring, long or slow 12 Technical problems or bugs 7 The screen was too small 6 It was too confusing/there was no familiarisation 6 You couldn’t see the Doctor 5 The second most common comment about the doctors questions was that the questions were too similar (8). This should be seen as related to the second most common complaint about the package generally which was that the options or paths that were available were either too similar or too limited. This raises two related issues. First, students found it difficult to discriminate between the types of questions that the Doctor was asking in 118 Australian Journal of Educational Technology, 2000, 16(2) the virtual interview; and second they therefore saw themselves as having few options with the kinds of questions they could ask. The issue of question similarity was mentioned by the content experts. One content expert thought that the “nuances in the way the questions were read out may not be of big enough difference for overseas students to pick up” while another said “some questions were too similar…students can’t make a choice”. The latter content expert suggested labelling questions to “aid learning and selection”. This may, however, undermine a fundamental goal of the package which is to assist students recognise and discriminate between different types of verbal questions and responses. By labelling questions students would effectively be told what type of verbal response is being given. This issue will be explored further in the discussion. Other common comments about the worst aspects of the package were technical issues related to sound and volume (16), the lack of a “back” or “rewind” option (16) and the difficulty with continuity (12). This latter comment is reflected in the statistic that one-fifth of students (20.9%) indicated they found the flow of information from screen to screen confusing. While this may be because of workbook distractions or technical difficulties it seems that the open structure of the package resulted in a lack of coherency in the “virtual interview”. The major problems seem to be that questions were revisited, the sequence of information in the interview was not logical for students, and the information revealed in the interview could be quite disjointed depending on the paths chosen. A number of students also found the package long, boring or slow (12). This may be related to the fact that many students found the package repetitious or that their questions or options were too similar. The program and the interface generally A number of questions were asked of students to determine their perceptions of the package generally. These questions (both open and fixed response) focussed on a number of issues including interactivity, navigation and functionality. An overwhelming majority of students thought the package had clear aims (73.7%), was interactive (76.4%), was easy to navigate around (74.6%), and had buttons and links that were easy Liaw, Kennedy, Keppell, Marty and McNair 119 to understand (85.4%). Students were also asked to rate the package and the interface generally and were overwhelmingly positive. Students found the package engaging, user-friendly and interesting. While some students found the package boring, most thought it was neither boring nor confusing. A summary of these responses is presented in Table 4 below. Table 4: Means scores and standard deviation for students ratings of the package and the interface generally. The package generally (‘1’ = not at all; ‘9’ = extremely) M SD Engaging 5.88 1.51 Appealing 6.03 1.58 Confusing 3.44 1.87 Interesting 6.31 1.60 User-friendly 6.93 1.54 Valuable 6.32 1.67 Boring 4.02 1.89 Enjoyable 5.95 1.56 Logical 6.16 1.80 The interface generally (‘1’ = poor; ‘9’ = excellent) M SD Useability 6.67 1.38 Clarity 6.57 1.62 Structure 6.05 1.83 Appeal 6.20 1.75 Another indication of students’ positive attitude towards the package (and the possible reasons for it) was from responses to the statement “Please note what you thought were the three best aspects of the package”. A summary of the ten most frequent responses is presented in Table 5. Aside from the audio and video, which has been mentioned above, students thought the interactivity, the options and paths available and the ease of use were the most positive features of the package. Other comments centred on the content and the ability to observe the patient’s body language. It is worth noting that some features of the package which were regarded as the best features by some students, were regarded as the worst features by others (eg. different options, the questions). 120 Australian Journal of Educational Technology, 2000, 16(2) Technical issues Other than the difficulties experienced with the audio, relatively few technical difficulties emerged. The extensive reviews conducted by the educational evaluator and the internal content expert in the previous stage of the formative evaluation probably accounted for the low number of technical difficulties. This notwithstanding, on two occasions the audio clip selected by the user did not correspond with the following video. This problem was noted by both content experts and students. Table 5: Highest ten ranked responses to the statement “Please note what you thought were the three best aspects of the package”. Response Frequency The video 46 The audio 36 The interactivity 23 The options available and the different paths 20 The user-friendliness or ease of use 20 The content: learning about communication and interview skills 15 The good design and presentation 13 Being able to see the patient’s responses and body language 11 The good questions 9 The chance to apply skills learnt in tutorials 8 Students’ understanding Finally, a preliminary evaluation of how the package assisted students’ understanding was carried out. The program of formative evaluation developed in the Faculty advocates that as a package approaches a beta version the assessment of learning outcomes, in some form, becomes more important. With this in mind, students were asked how the package helped them learn about communication and interview techniques. It should be noted that this is not an assessment of student learning, rather an assessment of students’ perceptions of how useful the package was in furthering their understanding. Students were complimentary about the helpfulness of the package with many seeing it as a tool which extended their knowledge while a smaller number felt it reinforced what had already been covered in their tutorials. This result is reinforced by the majority of students (80.0%) indicating that the package encouraged them to reflect on the content area. Other specific comments relating to students’ understanding are reported in Table 6. Only five students mentioned that Liaw, Kennedy, Keppell, Marty and McNair 121 their learning was limited by either the lack of feedback (3), continuity (1) or options (1). Table 6: Highest ten ranked responses to the statement “Please comment on the how the program helped you learn about communication and interview techniques”. Response Frequency My understanding of the use of questions, language and communication in general was improved 53 I didn’t learn much new; the program reinforced material from the tutorials 12 I had an improved idea of the general process of interviewing 7 I learnt more about observing body language and non-verbal cues 6 I had an improved understanding about dealing with Non English Speaking Background patients 4 The lack of feedback limited learning 3 It was much better learning than reading from a book 3 It improved my understanding of how to approach patients 2 It showed me the importance of listening 2 It showed me the importance of empathy 2 Discussion The alpha version formative evaluation of the package indicated that both content experts and students generally had very positive perceptions of the package. Student observation revealed that students had few troubles with the functionality (with the exception of volume control) and found the package appealing. Students reported that using the package was interesting, engaging, user-friendly and a valuable learning experience. The workbook was successfully integrated with the package, with the majority of students finding it easy to move between the workbook and the package. Many students also commented that the workbook helped them direct their learning. While there were problems associated with the use of audio, students and content experts alike validated the use of these media to create a realistic clinical interview situation. Finally, many students reported that the package helped them understand issues associated with communication and interviewing, suggesting the package served its intended purpose. Despite these positive findings, the evaluation highlighted a number of areas where either the package or the learning environment could be improved or modified. The primary recommendations that emerged from this evaluation, and the response of the development team to them, are outlined below. 122 Australian Journal of Educational Technology, 2000, 16(2) One recommendation which emerged from the alpha version evaluation related to the continuity, flow and sequence of information contained within the module. The finding that some students found the package repetitious and disjointed was of concern to the development team. These perceptions seemed to be fundamentally linked to the use of doctor’s questions within the package. It was recommended that the development team review their approach to the use of audio and video segments that underpin the doctors questions and the patients responses. It was expected that if modifications were made in how these segments were used, then the way in which students created their virtual interview could be enhanced, directly addressing the issues of continuity and repetition. Rather than dealing with relatively short segments of audio and video it was decided that longer audio-video segments, representing particular aspects of the doctor/patient interaction, be created. Each segment should make sense on its own (for example, a set of segments on sleep problems, diet or depression). By using larger segments it will be easier to organise the program so that repetition is avoided. Students will also be provided with access to a site map which will record their pathway through the interview. At any point in their interview students will be able to call up either a text or an audio-visual transcript of their virtual interview. Students will be able to review audio-video segments directly or they can access them via hyperlinks from a transcript of their interview. Thus, students will be able to use the site map to review their last few interactions or to review their entire interview. It is hoped that by providing this facility, students will reflect more generally on their interview, rather than on specific sequences. It is hoped that students will see continuity in their interview by using this function. Considering the almost infinite permutations in variations in nuance, style, words and non-verbal behaviour of both patient and doctor and the fact that we could not incorporate past experience into the model, it is an achievement that only a minority of students felt that they had limited options or pathways within the package. A difficulty that both content experts and students mentioned was the trouble some students had in detecting often rather subtle differences between the doctor’s questions. One content expert suggested labelling the questions may be a way to circumvent this problem. However, as mentioned above, specifying the nature of the questions would undermine a major thrust of the package, as it would effectively tell students what type of microskill is being employed by the doctor. Nevertheless, a number of other recommendations emerged from the evaluation. First, it was recommended that the development team take more care in the selection of the doctor's questions so that there is a clearer demarcation between question types. Second, the inclusion of a Liaw, Kennedy, Keppell, Marty and McNair 123 tutorial module in the CFL package was suggested which would give students audio and video examples of different types of microskills used in the interviews contained within the package. Through interactive tasks in the tutorial students could see more explicitly the advantages and disadvantages associated with using different types of questions and microskills in a variety of interview situations. Other than these recommendations, which were accepted by the development team, it also seemed worthwhile to find a solution to the problem of question discrimination which was embedded within the virtual interview itself. A possibility that the development team is currently considering is specifying the nature of the doctor’s question on the screen after students have made their initial, unguided question selection. By asking for an expert comment, students would be able to recognise the type of question they have selected on the previous screen and this information would help students appreciate the patient’s response. By incorporating “rewind” and “go back” functions, students would be more easily able to investigate the implications of asking different types of questions. While there were not too many negative comments about the interface, one content expert noticed that there was inconsistency in some of the functions of the buttons and links. The development team was aware that in the alpha version of the package there was inconsistency in whether buttons were rollovers, whether they were clickable or whether they were purely cosmetic. Some content experts expressed concern that the instructions which were given to students were not adequate, while many students requested that rewind or go back functions be incorporated into the design. The difficulties that were encountered with the audio quality indicated the need for a more user-friendly means of volume control. These results regarding the interface and graphic design led to a recommendation that the interface be redesigned. The development team, who were already planning this action, decided that the revised interface would incorporate a volume control, a rewind function and a go back function. The interface will endeavour to be more intuitive and will make a distinction between general program navigation and audio/video controls. One final, but crucial, criterion for the revised interface will be to employ consistent functionality of buttons. 124 Australian Journal of Educational Technology, 2000, 16(2) Conclusion The development team feels they have developed a CFL package that will assist students with their clinical communication skills and in developing an integrated biopsychosocial approach to patient diagnosis. The stages of formative evaluation this package went through were extremely helpful in affirming many of the design decisions that were made by the development team. Formative evaluation was also crucial in the process of refining the package and in offering suggestions for modifications in specific areas. The package is in the process of being developed further by incorporating more clinical scenarios. The program of formative evaluation reported here will lead into a beta version evaluation of students’ learning processes and products. Acknowledgments The authors would like to acknowledge invaluable work of those from the Biomedical Multimedia Unit, Ian Shiel, Russell Evans and the team from the Multimedia Education Unit, Peter Grace, and the actors. This multimedia project was funded from the Teaching and Learning (Multimedia and Educational Technology) Committee at the University of Melbourne. References Boud, D., Keogh, R. & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh & D. Walker (Eds.), Reflection: Turning experience into learning. (pp. 18-40). London: Kogan Page. Brown, Collins & Duguid (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-42. Herrington, T., Herrington, J. & Oliver, R. (1999). Providing reflective online support for preservice teachers on professional practice in schools. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (Ed-Media 99) (pp. 166-171). Charlottesville, AACA. Herrington, J. & Oliver, R (1995). Critical characteristics of situated learning: Implications for the instructional design of multimedia. In J. Pierce (Ed.), Proceedings ASCILITE'95. University of Melbourne, Australia. http://www.ascilite.org.au/conferences/melbourne95/smtu/papers/herrington.pdf [verified 12 Jul 2000] Herrington, J. & Oliver, R. (1997). Multimedia, magic and the way students respond to a situated learning environment. Australian Journal of Educational Technology, 13(2), 127-143. http://cleo.murdoch.edu.au/ajet/ajet13/su97p127.html Liaw, Kennedy, Keppell, Marty and McNair 125 Kennedy, D. M. & McNaught, C. (1997). Use of concept mapping in the design of learning tools for interactive multimedia. Journal of Interactive Learning Research, 8(3/4), 389-406. Kennedy, G. E. (1999a). Defining the dimensions of a formative evaluation program: A Multi-method, multi-perspective approach to the evaluation of multimedia. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (Ed-Media 99) (pp. 604-609). Charlottesville, AACA. Kennedy, G. E. (1999b). Critical reflection on a program of development evaluation. In J. Winn (Ed.) Proceedings ASCILITE'99, 165-171. Queensland University of Technology, Australia. [verifed 12 Jul 2000] http://www.ascilite.org.au/conferences/brisbane99/papers/kennedy.pdf Kennedy, G. E., Petrovic, T. & Keppell, M. (1998). A program of evaluation for multimedia technologies being developed in the Faculty of Medicine, Dentistry and Health Sciences. In R. M. Corderoy (Ed.), Proceedings ASCILITE'98, 407-415. University of Wollongong, Australia. [verified 12 Jul 2000] http://cedir.uow.edu.au/ASCILITE98/asc-98pdf/kennedypetrovickeppel.pdf Keppell, M. & Bennett, S. (1997). Digital Learning. A guide to multimedia and on-line development in the Interactive Multimedia Unit. Central Queensland University. Keppell, M., Elliott, K. & Harris, P. (1998). Problem based learning and multimedia: Innovation for improved learning of medical concepts. In R. M. Corderoy (Ed.), Proceedings ASCILITE'98, 417-424. University of Wollongong, NSW. http://cedir.uow.edu.au/ASCILITE98/asc98-pdf/keppellelliotharris.pdf [verified 12 Jul 2000] Koschmann, T., Kelson, A. C., Feltovich, P. J. & Barrows, H. S. (1996). Computer- supported problem-based learning: A principled approach to the use of computers in collaborative learning. In Koschmann, T. (Ed). CSCL Theory and Practice in an emerging paradigm. (pp 83-124). Erlbaum. Medical AudioVisual Communications Inc (1998). A practical guide to communication skills in clinical practice. CD-ROM, University of Texas & CPSO Canada. Schon, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books Inc. Teng Liaw, Department of General Practice and Public Health Gregor Kennedy, Biomedical Multimedia Unit Mike Keppell, Biomedical Multimedia Unit John Marty, Department of General Practice and Public Health Ruth McNair, Department of General Practice and Public Health Faculty of Medicine and Dentistry and Health Sciences The University of Melbourne