Microsoft Word - muller.doc Australasian Journal of Educational Technology 2005, 21(4), 491-509 Determining the factors affecting student perceptions of a popular science video Derek A. Muller and Manjula D. Sharma University of Sydney Video is a widely used resource in teaching at all levels of education, yet research regarding its pedagogic development and use is inconsistent, dependent on outdated learning theories, and inapplicable to the current cohort of students. This study aimed to determine the key multimedia design features, from the student perspective, related to learning from the popular science video, Dr Karl’s Falling Cats. The goal was to compare factors arising from three focus groups, each comprised of low, moderate, or highly interested students, with those outlined by current multimedia research. Overall, students’ preferences coincided with research in the areas of context, tone, colours, sound, and onscreen text. Participants in the moderate and high interest groups were more accommodating to complex aspects of the film that overwhelmed some novices, consistent with the literature. Although the design of Falling Cats was largely based on intuitive choices, it adheres closely to research based principles; this is identified as one of the keys to its success. Introduction Use of conventional video Video is the most widely used visual teaching technology besides the overhead projector. It has been a fixture in classrooms of all levels since the late 1980s when over 90% of schools had at least one video player and over half had more than one (Glenn & Carrier, 1989). By the mid 1990s video facilities were available in practically all educational settings (Wetzel, Radtke & Stern, 1994, p.22; see also: Oliver, Grant & Younger, 1994). A more recent study of video use in higher education found the majority of faculty members in all departments view video as “extremely useful”. The technology is seen as a clearly established and frequently used resource in tertiary teaching (Barford & Weston, 1997, pp.48-49). The obstacle to its more widespread use is not lack of perceived effectiveness, but the 492 Australasian Journal of Educational Technology, 2005, 21(4) technical difficulty involved in finding appropriate videos and displaying them effectively for all students. Not only is video playing an important role in education today, it continues to be integrated into new multimedia presentations on CDs and in online tutorials, linked to other hypermedia and downloadable in streaming format. Previous research on video The history of research into video in education is incongruous and contradictory due to several confounding factors. Firstly, the focus of early research was on validating film, television and video as teaching media, by comparative studies with more traditional methods of instruction, such as lectures, textbooks, and teaching laboratories. Instead of exploring the methodological potential of various media, researchers were looking “for evidence of media effectiveness” (Russell, 1985, p.47). Secondly, very few studies were based on a theoretical framework of why the interventions should work, and when they were, the learning theories used were dependent on a delivery model of education (Kozma, 1994a; Clark & Estes, 1999). Thirdly, the range of educational experiences possible with video is so immense that it is impractical to even consider making generalisations about the medium as a whole. The combination of these three factors led to conflicting results, divorced from theory, promoting scepticism among practitioners and educational administrators alike. Regardless of the outcomes of educational research, video technologies crept into classrooms at all levels and have remained there in spite of a lack of concrete research findings. This gradual rise to prominence of video media was dubbed a “quiet technological revolution”, as researchers at the time were focussing on computers, the next technological invention hyped to transform education (Reider, 1984, p.12; Roberts, 1993). Video technology has developed and been applied extensively over the past century, increasingly in educational contexts; however, it still frustrates designers, practitioners and administrators with its unpredictable successes and failures. The reason for its variable achievement has been identified by numerous researchers, notably Clark (1983, 1994), Kozma (1991, 1994a, 1994b), and Mayer (1997, 2001). The consensus reached by these researchers is that “with few exceptions, there is NOT a body of research on the design, use, and value of multimedia systems” (Moore, Burton & Myers, 2004, p.997; emphasis in original). This is not to suggest that research be undertaken to establish the effectiveness of the medium as a whole (as has been proven futile in the past), but to research how “instructional treatments affect the cognitive processing Muller and Sharma 493 within the learner”, and how this can be exploited to consistently achieve high learning gains (Clark, 1994, p.7). Clark and Mayer have suggested different strategies for overcoming the lack of research based evidence surrounding multimedia development. Clark believes that the void created by insubstantial research has been filled by ‘craft’ solutions; these are “limited, contextualized, non- transferable … solutions to educational problems”, (Clark & Estes, 1999, p.5) that rely heavily on a designer’s intuition and artistry. To move from these ‘craft’ technologies to more scientifically grounded ‘authentic educational technologies’ (see: Clark & Estes, 1998, 1999; Estes & Clark, 1999), Clark emphasises the need for evaluation, in the form of Kirkpatrick’s (1994) four principles. In brief, these are: 1) reaction, gauging participants' perceptions of the intervention; 2) learning, measuring changes in participant knowledge, skills, or attitudes; 3) behaviour, ensuring learning is put into practice; and 4) results, the bottom line outcomes of training, most applicable in business settings. Clark argues that this evaluation must be used both to investigate successful craft technologies and to validate future multimedia interventions. Mayer’s research program has involved fifteen years of theoretically grounded studies, systematically establishing principles of good practice in multimedia design with conclusions supported by empirical evidence (Mayer, 2001; R. C. Clark1 & Mayer, 2003). Mayer’s evidence of learning is usually in the form of retention and transfer tests. While these tests effectively address Kirkpatrick’s second evaluation principle, they ignore the first. Kirkpatrick notes that although later principles may seem more significant, all stages of the evaluation are equally important and necessary. Goals of the study The purpose of this study was therefore to perform Kirkpatrick’s first level of evaluation on a popular science video, comparing, where possible, Mayer’s (and others’) principles to student perceptions of relevant aspects of the film. These are not necessarily expected to be coincident as Clark (1982) pointed out: learners may not always prefer the instructional method that is most effective for their learning (see also: Lewis, 1995). However, in describing an educational intervention, learners are most likely to raise issues that were most memorable and enjoyable for them, which should therefore correlate with retention. The aim of this paper is twofold: firstly to report on how students’ preferences map onto Mayer’s principles, connecting the first two levels of Kirkpatrick’s evaluation; and 1 To avoid confusion, Ruth Colvin Clark is referred to as R. C. Clark throughout this paper, whereas Richard E. Clark is referred to simply as Clark. 494 Australasian Journal of Educational Technology, 2005, 21(4) secondly, to investigate the aspects of Falling Cats that are perceived as effective by students, to inform future developments of similar technologies. Research design Dr Karl’s Falling Cats Figure 1: Screen shot from Falling Cats Every year, students of first year physics at the University of Sydney are shown a five minute animated video by popular scientist Dr Karl Kruszelnicki (ABC, 2005) as an introduction to the topic of terminal velocity (Kruszelnicki, 2002). The film outlines the basics of terminal velocity to answer the question of why it is sometimes safer for cats to fall from taller buildings than lower ones. The film is based on data collected by New York City veterinarians on the injuries sustained by cats after accidental falls (Diamond, 1988, 1989). Narrated by an animated Dr Karl and starring an expressive orange tabby cat, the film takes a humorous look at some serious physics. Concepts like gravity and wind resistance are clearly illustrated to explain why falling bodies have a speed limit. The film has been shown on television and has achieved international recognition (Embassy of France in Australia, 1997). Lecturer perceptions Part of the impetus for this study arose from casual conversations with first year lecturers who used Falling Cats in their teaching. Lecturers believed showing the video promoted interest in the subject matter and captured the Muller and Sharma 495 attention of students, carrying over to the remainder of the lecture. They also felt watching the video was genuinely enjoyable; an important aspect of higher education that they thought was often overlooked. These views around teaching with video are common among tertiary lecturers (Oliver, Grant & Younger, 1994, p.29). Initial findings Evidence of the depth of student understanding of terminal velocity after watching the video has already been compiled and analysed (Muller & Sharma, 2005). Results showed that the high and moderate interest groups demonstrated the deepest understanding of the physics, as measured by the number of correct concepts mentioned, the confidence with which these ideas were raised, and the expression of independently developed opinions of the physics. This analysis was based on a model for science communication, measuring learning on the dimensions of Awareness, Enjoyment, Interest, Opinion forming, and Understanding (the vowel analogy, see Burns, O’Connor & Stocklmayer, 2003). A more interesting finding of this analysis was the willingness of students in all focus groups to ask questions about the physics in the video without being prompted. Comments illustrating participant learning were excluded in the data set for this analysis, as the focus is explicitly on teasing out the video’s key attributes from the students’ perspective, not on their learning. Method Three focus groups of university students were assembled on the basis of prior teaching and overall interest in physics: low (n=5), moderate (n=8), and high level (n=18) groups. Students who self reported having a lower interest in physics had less prior physics instruction and vice versa. Interest in physics was self reported on a five point scale from very low (1) to very high (5). Students in the low, moderate, and high interest groups averaged scores of 2.2, 3.3, and 4.3 respectively. Students in the moderate and high interest groups were first year students and had taken physics in first semester. The low interest group was selected from a range of university levels with little prior physics instruction besides compulsory high school science. The above focus group characteristics are summarised in Table 1, below. A similar number of students was expected to participate in each group. However due to the nature of the study, students with greater interest in physics volunteered more readily. Participants were all between the ages of 17-27 and were enrolled in undergraduate studies at the University of Sydney. 496 Australasian Journal of Educational Technology, 2005, 21(4) Table 1: Summary of focus group attributes Low interest group Moderate interest group High interest group Experience in physics High school or less First year fundamental First year advanced Average self-reported interest (1 = very low, 5 = very high) 2.2 3.3 4.3 Number of participants 5 8 18 All focus groups met for approximately one hour. This included watching the video, independently filling out a short answer questionnaire, and, as a group, discussing the results. The discussions were recorded and later transcribed. The questionnaire contained six broad, opinion based questions for discussion, and one question on the participant’s personal definition of terminal velocity. The goal of the questionnaire was to get participants thinking about the issues to be discussed in the focus group and to obtain independent opinions of the video prior to the discussion. The qualitative method of focus group research was selected because it most effectively met the criteria for the study. Focus groups are known to yield rich data in a relatively short period of time (Greenbaum, 1998). They are often used in commercial evaluations of videos and products, and allow students to watch the video in a social setting, more analogous to a lecture. Results and analysis Discussions from the focus groups were transcribed and coded using open and axial coding as described by Strauss & Corbin (1998). Although this grounded theory technique is typically used for theory generation, the selective coding and theory generation steps were omitted as they are beyond the scope of this study. In open coding, all statements in the transcripts were broken down into components using ‘in situ’ codes; these are phrases taken verbatim from the data. In the axial coding step, the data broken down in open coding is synthesised by combining relevant data into categories. A single comment is included in all categories to which it pertains. These groupings are then arranged into higher order categories, and iterative comparisons between the data and higher categories are made to ensure fidelity to the observations. Where quotes are given to support generalisations from the data, participants are referred to by an arbitrary code. Codes beginning with L, M, and H refer to participants from the low, moderate, and high interest groups respectively. The investigator is referred to using the letter ‘I’. Muller and Sharma 497 0 5 10 15 20 25 Ag e gr ou p Ph ys ics co nt en t Co nt ex t In te re st Ex pl an at io ns Hu m ou r To ne Co lo ur s So un d Te xt Le ng th En jo ym en t Factor Frequency High Interest Moderate Interest Low Interest Figure 2: Frequency of comments pertaining to factor categories Figure 2 shows the final categories and the relative frequencies with which they emerged in each focus group. Each of these categories is addressed below, indicating the way in which the factor appeared and how this compares with relevant research. Age group In all of the focus groups, the target age group of the video emerged as an important issue. Specifically, all participants felt that Falling Cats was appropriate for younger viewers, typically between the ages of twelve and sixteen. This opinion had important implications for the ensuing discussions as participants adopted different views of their roles in the focus groups. Some students saw themselves as authorities on learning, based on their experience at school, and commented on how younger students would view the video. Others viewed themselves as the target audience and responded to questions as though they’d seen the video in a lecture. Sometimes these two perceptions were discussed explicitly. M1 They didn’t really explain what it was all about. I think there was more on cats than on actual terminal velocity. M2 There’s books, there’s lectures… M1 I know, but if you’re going to make it about it - M2 Depends what the age group it’s for, though. For us we want to know more about it. Whereas for younger audiences… It introduces them to it, so it just depends on the age group. 498 Australasian Journal of Educational Technology, 2005, 21(4) The focus on the target age group of the video was unanticipated, as Falling Cats is commonly shown to first year students. However, this topic often resulted in contrasting opinions: H1 I suppose maybe it’s because it’s probably for a younger audience, but [it’s] just that [the video] kind of talked down to the audience. In a separate instance, a participant in the same focus group remarked: H2 I know for younger audiences it would help a lot. I don’t know if it’s just that I’m really immature but I’d prefer to learn from that than listening to a lecturer at the front of a lecture theatre. To prevent these two viewpoints from polarising results, comments specifically pertaining to a younger age group were classified in the age group category and were not included in the other categories. In research literature, animation and humour are two aspects of the video that have been shown to be more successful techniques for younger audiences than for older learners (Wetzel, Radtke & Stern, 1994). This could have contributed to participants’ perceptions that the video would be more appropriate for younger viewers. The consensus reached by the vast majority of participants was that the video was targeted at younger viewers but could be used as a starting point for university students. One exception is given below: I And you don’t think it’s worth showing it to people who are older but who haven’t taken physics before? What is your feeling on that? L1 I think no, probably not. I think they could do something similar but for older people. When asked what would improve the video for an older audience, this participant responded: L1 More physics, something more complicated but the same style. Physics content This desire for more physics to be included in the video was expressed by participants in all focus groups. As shown in Figure 2, this theme, unsurprisingly, emerged twice as strongly in the moderate and high interest groups compared with the low interest group. Although it was dependent on the target age group, in general participants felt the video “should be more information dense.” (L2) This was not necessarily a negative reflection on the film, however, since participants believed they could learn more from the video. M3 I think it should have gone more in depth about terminal velocity. M4 Yeah, could’ve gone a bit longer. Muller and Sharma 499 M3 I think, cause it was interesting…. M5 You’re just getting into it and then it’s done. Because the film caught their interest, participants felt they could pay attention and learn more if the video explored the physics to a greater extent. Some participants in the high interest group, however, doubted that the physics could be explained in enough depth with only the use of the video. H3 Well, it was good in the fact that it tells you what the topic’s about, but it’s bad that you don’t have … firm maths behind it as well. So it’d be a good introduction, but whether it would work during the lecture… Context Two different ideas of context arose strongly in the focus groups: 1) the use of an animated video for teaching, and 2) the use of the Falling Cats narrative to situate ideas about terminal velocity. Animated video Participants uniformly agreed that the animation provided a unique and effective context for learning about terminal velocity. M6 I think it’s a different approach. It’s a different approach to … M7 Other physics teachers. H4 If you can take that and put in university subject matter, then it would be a pretty good way to learn. As is evident from the quotes above, participants related to and enjoyed learning in the animated video context. They often contrasted this type of learning with what they experienced in lectures, viewing this approach as more innovative and interesting. The use of animation in illustrating physics concepts is in line with the studies reviewed by Wetzel et al. (1994) and the principles suggested by Rieber (1990). Wetzel et al. found that animations were most effective for younger students, possibly contributing to the age group issue discussed above. Rieber found that animation should only be used when its purpose is related to visualisation, motion or trajectory. These attributes are particularly relevant to the explanation of falling cats and terminal velocity. Falling Cats narrative Participants also enjoyed many aspects of the storyline and narration. They thought the use of a cat to demonstrate the physics allowed them to better relate to the subject matter. They also found details of the cat narrative and interwoven real world facts intriguing and beneficial to their learning. 500 Australasian Journal of Educational Technology, 2005, 21(4) M8 I thought the fact that they used falling cats instead of something boring like a ball was a good idea. I think using something in real life was a good idea, because you can relate to it. M9 It’s an indirect way of teaching people, because it’s not based on terminal velocity, it’s based on cats. Not only did the context situate the physics ideas of terminal velocity in a quasi-realistic setting, it added an affective component that caught students’ attention. M10 I liked it how Dr Karl’s cartoon face popped up occasionally as the narrator and the cat would sort of be like huh? (laughter) Yeah, that was funny, engaging. R. C. Clark and Mayer warn against using attention grabbing devices as these can bombard learners with extraneous information, making it more difficult to select the salient points from an intervention. Falling Cats is different, however, from the examples described by these researchers as the physics is integrated into the interesting context. Therefore, the narrative gives learners a structure on which to organise their knowledge, which has a positive impact on learning as shown by Mayer (2001). R. C. Clark and Mayer have also demonstrated the utility of onscreen coaches or pedagogical agents; these are tutors that appear in a multimedia presentation to guide the learner. They argue that use of these onscreen personalities promotes student interactions with the technology in a more natural, social way that is more conducive to learning. Their research has shown that students perform better on tests of retention and transfer with the use of these agents than without. Interest As the lecturers predicted, a key aspect of the video was its ability to inspire interest. This interest was expressed both towards the video and towards the subject matter in general. Students cited several reasons why they were interested in the video, ranging from animation techniques to the narrative context. M11 With the sounds and colours it was more interesting than just a boring video tape - you’d fall asleep. It was really jazzed up so you want to see what’s going to happen. M12 You’re sort of interested because you want to know how many cats died… Participants often voiced the belief that the video could make the physics more attractive; one student commented, “It would just be a quick introduction, to get [students] interested.” (H5) From this introduction, participants believed the lecture would be more effective. Muller and Sharma 501 H6 It gives you a basic idea of the concepts, and then maybe you’d be better able to understand [the lecture] than without this intro. Not all participants believed that the increased interest resulted in more learning, however, as the following quote demonstrates: L3 The video would hold my interest more, but I’d probably in the long run learn more in the lecture, because though I wouldn’t pay a lot of attention at the time, I’d write down a lot of notes, so… This student doubts the informational content of the video or believes she’d better be able to recall the lecture by note taking. Studies have shown that students view televised instruction as effortless and ineffective for learning due to its close relationship to entertainment (Krendl, 1986). Explanations The explanations of the physics in Falling Cats were regarded by the moderate and high interest groups as very effective. Participants commonly identified the clear, simple language employed in the narration, supported by explicit visuals as the reason the physics material was easy to follow. They also felt that presenting the material in both verbal and visual streams was a beneficial learning technique. H7 It used video explanations in conjunction with talking so you’re learning the same thing in two different ways. Mayer (1997, 2001) has repeatedly shown that learning with text and pictures or animation and narration is more successful than with any one mode alone. This finding is based on Paivio’s dual coding theory (1971, 1986), which says that the brain processes verbal and non-verbal information separately. When the information in these two streams is part of the same message, linkage formation between verbal and non-verbal models significantly increases the probability of recall and transfer. In the low interest session, the explanations received a mixed response. Again some respondents felt the physics was presented clearly. L4 They wouldn’t just say that gravity is this thing and give you a formula, but they explain it to you in a way that you know what they mean, and show examples. Others, on the other hand, felt the details given in the video lacked the rigor necessary to support the physics assertions presented. L5 I thought some of the concepts that I wasn’t familiar with, I would have liked to see explained. I don’t really understand dynamic tension. I also felt the details were anecdotal, like I don’t necessarily feel I’ve learned some facts I can use or rely on. 502 Australasian Journal of Educational Technology, 2005, 21(4) This suggests that the video explanations were less accessible to the low interested students due to their lack of familiarity with physics material. The difficulty that novices encounter in learning from presentations that are extremely useful to experts has been documented in many different contexts (e.g. Lowe, 2003, 2004). Humour Participants in all focus groups addressed the humour in Falling Cats before discussing most other aspects of the film. Despite some commenting that the jokes were not targeted at the university age group, humour was appreciated by the majority of students, who felt it also helped them learn. L6 I just thought that the humour was very efficient. I Efficient? L6 Yeah, as a means of teaching. Humour was mentioned by participants in all groups as one of their favourite aspects of the video. The only contentious point surrounding humour was to do with the use of puns. Some students thoroughly enjoyed them, “I loved the puns” (M13); whereas other students thought they were tacky and inappropriate. Whether liked or disliked, however, the puns elicited a strong reaction from the focus groups. In reviews of related literature, the consensus reached is that humour is most effective for younger viewers with decreasing impact as age increases (Zillmann et al., 1984). “Only humour that is well integrated with educational materials is likely to enhance student-teacher rapport and does not produce the negative effect on learning.” (p.802) These findings may not be applicable to the new cohort of students, however, as the culture of entertainment and learning has changed drastically in the past twenty years. Adult cartoons are much more common in entertainment and students increasingly expect to be entertained in lectures (Altschuler, 1999). Since there are ever expanding avenues through which students can find information outside the lecture (McInnis, 2000), lecturers must find innovative ways of presenting material or risk declining attendances (Stevenson, 2005). Tone Participants related strongly to the tone of the video, specifically the lively, colloquial narration by Dr Karl. They felt this attracted them to the video. M14 The tone of voice. It wasn’t monotone, like it was… M15 It was engaging… H8 It was pretty easy-going and it didn’t take itself too seriously - kind of made it fun. Muller and Sharma 503 This topic also lead to comparisons with physics lecturers as students felt the tone contrasted strongly with what they experienced in class. Of Dr Karl, one student remarked, “he’s not some droning lecturer.” (H9) This point is supported by the results from R. C. Clark & Mayer (2003), who found that using personal rather than formal language promotes learning. They believe the use of casual language in the first or second person encourages more natural social interactions with technology, analogous to the effect of using a pedagogical agent. This is supported by empirical evidence that showed learners achieved higher scores on transfer tests after watching personalised interventions than after watching more formal ones. Colour The colours used in the video were another aspect of the animation mentioned in all focus groups. This is an area in which the video design departs from research findings and adopts a more intuitive approach; this is one of the ways in which the video is a ‘craft’ solution. The video used a wide palette of bright colours to create complicated and colourful scenes and backgrounds. Most participants agreed this added to the visual appeal. M16 I thought it was really colourful. Just the colours, really – it comes at you, but in a good way. Only one student in the low interest group commented on the colours, finding them distracting: L7 I think at times it was a bit hectic. Cause there were too many colours and it changed all the time. Despite a small increase in learning with coloured as opposed to black and white media, the main recommendation from the literature is that colours should be limited and allow for clear discerning of objects (Wetzel, Radtke & Stern, 1994). Practitioners like Blinn (1989) agree with this assertion, arguing the essential objects in the animation must stand out and background details must be minimised. It is possible that some negative student perceptions around colour use in this video were induced by some loss of clarity in the video image due to VHS dubbing. One student in the high interest group may have picked up on this: H10 I felt that since it is a visual medium for educating people, the colors used were all reds and light purples and all that, so sort of confusing that – sort of images were blurred so you couldn’t really bring out where a cat was in between the sky. 504 Australasian Journal of Educational Technology, 2005, 21(4) Some studies have shown that a loss in pictorial clarity can result in detrimental effects on students’ perceptions and learning from a medium (Wetzel, Radtke & Stern, 1994). Sound The sound in Falling Cats is layered throughout with several musical backgrounds, narration, cartoon sound effects, the cat’s vocalisation, and ambient street noise. This choice again clearly illustrates the ‘craft’ nature of the intervention as it goes against research findings that suggest sounds should be kept to a minimum to promote learning (R. C. Clark & Mayer, 2003; Mayer, 2001). Previous findings also showed adding sounds “as background noise or to add realism to narrative” have no effect on learning (Barrington, 1970, 1972, as cited in Wetzel, Radtke & Stern, 1994, p.143). Participants in the low interest group reacted negatively to the sound in the video. They felt the sounds were sometimes distracting or that the music didn’t seem appropriate. L10 I didn’t like the music in the beginning because it made too much of an effort like OK, let’s do something that young people like. In the moderate interest group, reactions were mixed. M17 I said the music, the sound effects, and also the fast cartoon - I thought it might distract kids a bit from Karl’s speech. Others thought that the sounds attracted their attention and made the video more enjoyable. M18 I thought like the music and all that, was like - I actually had that as one of my likes, because it kept me interested. This was a thought echoed by the high interest group, who felt the sounds and music added to the humour of the film. The moderate and high interest groups may have felt more positively about the sounds than the low interest group because they have more experience with the physics and were therefore less likely to feel overwhelmed or distracted by inconsequential details (Mayer, 2001). Text One aspect that was appreciated by participants in all three groups was the use of large bold animated text that appeared on screen every time a new topic area was introduced or a concept was explained. Muller and Sharma 505 M19 One thing I did like was that they reinforce all the main points with big bold writing. H11 Topic headings like how he goes ‘this is the realm of fact but now we’re moving into the land of theory.’ Mayer (2001) showed that use of on screen text in conjunction with pictures and narration had a detrimental effect on learning, as it split the attention of learners between the visuals and text. He suspected, however, that use of on screen text when not in competition with other visuals might yield different results, and recommended further studies be undertaken. This is a point supported by Borzyskowski (2004), who argues that literacy is more a basic skill than the ability to comprehend complicated visuals. Therefore, animated text may be the intermediate step between text and visual representations necessary to scaffold novices and learners from one medium to the next. Length In the moderate and high interest groups, participants agreed that the video was short and concise. M20 It was quick. Like it just didn’t drag on. M21 Yeah, it was straight to the point. This opinion had both positive and negative implications, however, as some found the video succinct, while others found it lacking in substance. This led to suggestions that more physics content be included, as outlined above. Since the majority of teachers use videos “as a supplementary part of their teaching strategies and not as an alternative to them” (Barford & Weston, 1997, p.46) the video is an appropriate length for viewing in lectures. Additionally, the finding that students wanted more physics from the video suggests that lecturer reviews and additions after the film would be an effective method of teaching. Introductions and follow up discussions have been shown to increase learning, sometimes more than a second viewing of the film (Wetzel, Radtke & Stern, 1994; Russell, 1985). Discussion In this study, the first level of Kirkpatrick’s evaluation was performed on the popular science video, Falling Cats, using three focus groups of students with varying physics backgrounds. From the transcripts a series of categories were developed, which were then used to compare student preferences to multimedia design principles. In addition, students’ 506 Australasian Journal of Educational Technology, 2005, 21(4) comments were analysed and can applied to the development of future educational interventions since “many successful technologies have resulted from descriptive reasoning about why the craft solution worked.” (Clark & Estes, 1999, p.10) In general, students’ preferences matched up surprisingly well with prominent multimedia design principles. Participants were positive about aspects of the video, such as the casual tone, use of an onscreen pedagogical agent, and context on which to organise knowledge, supported by research based findings. Reactions were mixed, however, to the colour and sound in the video, which were more complicated than studies have shown to be effective. Humour was one aspect that has been shown to have little effect on learning for older students but was strongly liked by the majority of participants in all focus groups. This may be due to the way in which humour was well integrated into the subject matter or due to the changes in student culture over the last twenty years. The results of this study were complicated by two factors: 1) the emergence of the target age group as an important issue in all focus groups, and 2) the overwhelming of novice learners by sounds, colours, and anecdotal evidence. Due to conflicting opinions on the target age group of the video, participants either adopted an expert stance or viewed themselves as the target group of the intervention. This led to contrasting opinions of various aspects of the film. The low interest group was more negative about the context, explanations, colours, and sounds in the film, likely due to their unfamiliarity with the subject matter. Students, regardless of background or intrinsic interest in physics, wanted to know more about the physics after watching the video. They also wanted more physics content to be included in the video style. This suggests that Falling Cats is effective in the manner perceived by lecturers. The factor of context is multi-faceted in that it can be realistic or fantastical, and well integrated or an intermittent diversion. As this topic evoked a strong reaction from participants, further research in this area should aid in the development of multimedia with a positive influence on both learner perceptions and test scores. The categories outlined in this study cover a broad range of aspects that were important to students and would be vital to consider when designing future educational videos. This study also highlights the need to evaluate educational interventions, not only when they are first developed and introduced, as is often the case, but when they have been selected by lecturers or used routinely. Muller and Sharma 507 Conclusions The goal of this study was to qualitatively evaluate a popular science video to: 1) compare student preferences with research based principles of multimedia design, and 2) identify the keys to its success from the student perspective. Participants in all focus groups reacted positively to the majority of aspects of the video that adhered to design principles. This adds a dimension to research based studies identifying multimedia design principles that positively impact on student learning, by demonstrating these guidelines also have attitudinal benefits. Students appreciated the humour and context in the video, and enjoyed watching and learning from it. This emphasises the need for good integration of context or interesting material with educational content in multimedia design to promote positive student perceptions. Acknowledgements We are greatly indebted to Dr Karl (ABC, 2005) and Deborah Szapiro of Freerange Animation for graciously allowing us to use the Falling Cats video in this study. Thanks also to the students who volunteered their time and participated enthusiastically in the focus groups. References ABC (Australian Broadcasting Corporation) (2005). All about Dr Karl. http://www.abc.net.au/science/k2/aboutkk.htm Altschuler, G. C. (1999). Let me edutain you. The New York Times, 4 April, p. 50. Barford, J. & Weston, C. (1997). The use of video as a teaching resource in a new university. British Journal of Educational Technology, 28(1), 40-50. Blinn, J. (1989). The making of The Mechanical Universe. Paper presented at Spatial displays and spacial instruments: Proceedings of a conference sponsored by NASA Ames Research Center, and the School of Optometry, University of California, Asilomar, California. Borzyskowski, G. (2004). Animated text: More than meets the eye? In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 141-144). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/borzyskowski.html Burns, T. W., O'Conner, D. J. & Stocklmayer, S. M. (2003). Science communication: A contemporary definition. Public Understanding of Science, 12, 183-202. Clark, R. C. & Mayer, R. E. (2003). E-Learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (1st ed). San Francisco: Jossey-Bass/Pfeiffer. Clark, R. E. (1982). Spanning the gap between theory and practice. Instructional Innovator, 27(2), 16-18. 508 Australasian Journal of Educational Technology, 2005, 21(4) Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-29. Clark, R. E. (1994). Media and method. Educational Technology Research and Development, 42(3), 7-10. Clark, R. E. & Estes, F. (1998). Technology or craft: What are we doing? Educational Technology, 38(5), 5-11. Clark, R. E., & Estes, F. (1999). The development of authentic educational technologies. Educational Technology, 39(2), 5-16. Diamond, J. (1988). Why cats have nine lives. Nature, 332, 586-587. Diamond, J. (1989). How cats survive falls from New York skyscrapers. Natural History, August, 20-26. Embassy of France in Australia (1997). Images of science: Falling cats and Australian museums. French Science and Technology, No. 28. [viewed 26 Aug 2004; not found 28 Oct 2005] http://www.ambafrance- au.org/fst/pages/fst28/fst28p16.en.htm Estes, F. & Clark, R. E. (1999). Authentic educational technology: The lynchpin between theory and practice. Educational Technology, 39(6), 5-13. Glenn, A. D. & Carrier, C. A. (1989). A perspective on teacher technology training. Educational Technology, 29(3), 7-11. Greenbaum, T. L. (1998). The handbook for focus group research (2nd ed.). Thousand Oaks, California: Sage Publications. Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61(2), 179- 211. Kozma, R. B. (1994a). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7-19. Kozma, R. B. (1994b). A reply: Media and methods. Educational Technology Research and Development, 42(3), 1-14. Krendl, K. A. (1986). Media influence on learning: Examining the role of preconceptions. Educational Communication and Technology, 34(4), 223-234. Kruszelnicki, K. (2002). Falling cats. In Freerange Animation, Great Moments in Science. Australia: Australian Broadcasting Corporation. Lewis, R. A. (1995). Video introductions to laboratory: Students positive, grades unchanged. American Journal of Physics, 63(5), 468-470. Lowe, R. K. (2004). Interrogation of a dynamic visualization during learning. Learning and Instruction, 14(3), 257-274. Lowe, R. K. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157-176. Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels (1st ed). San Francisco, CA: Berrett-Koehler, Publishers Group West. Mayer, R. E. (2001). Multimedia learning. Cambridge, UK; New York: Cambridge University Press. Muller and Sharma 509 Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32(1), 1-19. McInnis, C., James, R. & Hartley, R. (2000). Trends in the first year experience in Australian universities (pp. 74). Melbourne: Department of Education, Training, and Youth Affairs. http://www.dest.gov.au/archive/highered/eippubs/eip00_6/fye.pdf Moore, D. M., Burton, J. K. & Myers, R. J. (2004). Multiple-channel communication: The theoretical and research foundations of multimedia. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 979-1005). Mahwah, NJ: Lawrence Erlbaum. Muller, D. A. & Sharma, M. D. (2005). Video physics education: Falling cats and terminal velocity. Paper presented at the Australian Institute of Physics: Physics for the Nation, Canberra. Oliver, R., Grant, M. & Younger, G. (1994). The Perth Educational Television Project. Australian Journal of Educational Technology, 10(1), 27-40. http://www.ascilite.org.au/ajet/ajet10/oliver.html Paivio, A. (1971). Imagery and verbal processes. New York: Holt. Paivio, A. (1986). Mental representations: A dual coding approach. New York and Oxford [Oxfordshire]: Oxford University Press; Clarendon Press. Rieber, L. P. (1990). Animation in computer-based instruction. Educational Technology Research and Development, 38, 77-86. Roberts, G. (1993). Educational technology and the mass lecture: A restatement of fundamental issues. Australian Journal of Educational Technology, 9(2), 182-187. http://www.ascilite.org.au/ajet/ajet9/roberts.html Russell, A. L. (1985). How media materials are used in university instruction: Reports of selected university faculty members. Australian Journal of Educational Technology, 1(1), 47-58. http://www.ascilite.org.au/ajet/ajet1/russell.html Stevenson, A. (2005). Empty lecture halls in the online campus. Sydney Morning Herald, 26-27 February, p.4. Wetzel, C. D., Radtke, P. H. & Stern, H. W. (1994). Instructional effectiveness of video media. Hillsdale, NJ: Lawrence Erlbaum Associates. Zillmann, D., Masland, J. L., Weaver, J. B., Lacey, L. A., Jacobs, N. E., Dow, J. H., et al. (1984). Effects of humorous distortions on children's learning from educational television. Journal of Educational Psychology, 76(5), 802-812. Derek A. Muller and Manjula D. Sharma Sydney University Physics Education Research Group School of Physics, University of Sydney, NSW 2006, Australia Corresponding author: Derek A. Muller. Email: muller@physics.usyd.edu.au