Microsoft Word - clarkson.doc Australasian Journal of Educational Technology 2007, 23(2), 248-268 Achieving synergies through generic skills: A strength of online communities Barnard Clarkson and Chris Brook Edith Cowan University Educators are often torn between impositions of the institution in which they work and the imperatives of their individual courses or units and the impact this tension might have on student satisfaction with the learning experience. It is common to hear that students must graduate with multiple generic attributes or skills, yet these skills may not be within the gamut normally required in a specific undergraduate unit. This paper reports on an attempt to integrate both University sanctioned or top down generic skills and an instructor’s organic or bottom up desirable skills in a multimedia unit at an Australian university, and the impact this has on student satisfaction. Specifically both asynchronous and synchronous tools were used to facilitate online community characteristics, in turn usable to foster the generic skills of collaboration, communication and problem solving. Results reveal synergies between the possibly divergent and potentially opposed goals of the university and the classroom. This paper demonstrates the ways that the conscious promotion of an online community to simultaneously assist achieving both the unit learning outcomes and prescribed generic skills, caused no evident conflict for student participants. Introduction In Australia many universities seek to ensure that all graduates should graduate with generic skills like collaboration, communication and problem solving, as well as the outcomes and processes of their substantive discipline. Although valuable, such outcomes may be seen as only peripheral to the requirements of any specific course or unit and may therefore compete with the individual competencies required in specific undergraduate units. How can units be structured so that they meet both institutional and classroom goals? If this is possible, will the unit still meet student expectations? The authors sought to identify synergies between the various goals of a tertiary web design unit and then identify online tools that would support this approach. The research goal was developmental and mixed research methods were used (Reeves, 1999) to provide evidence Clarkson and Brook 249 of student satisfaction with their ‘generic’ collaboration, their problem solving skills, and their satisfaction with a unit attempting to integrate system mandated generic skills (sometimes named 'graduate attributes'). The methodology for this research is described in some detail after a brief review of relevant literature, followed by the findings and conclusion. Overall the value of case studies such as this is improved understanding rather than simple generalisability (Mills, 2003); nevertheless the findings may prove relevant to similar units. Educators are increasingly being asked to utilise more constructivist, ‘authentic’ and ‘self directed’ approaches in efforts to increase the quality of their student’s learning. This requires the instructor to possess ‘an essential trust in the capacity of others to think for themselves and to learn for themselves’ (e.g. Rogers, 1983). At the same time there is greater pressure from universities to use more online settings (Segrave, Holt & Farmer, 2005), for example to justify the huge ICT costs and meet other institutional aims (James, McInnis & Devlin, 2002). The aim of this case study was to attempt integration of these varying and sometimes ‘opposing’ pressures. We were of the view that synchronising these pressures was a productive approach for educators, whatever their setting and for students, whatever their stage. Review of literature Contemporary research supports the authentic assessment and situated learning approach (Luca & Oliver, 2003). Notwithstanding traditional or ‘chalk and talk is best’ views (e.g. Bain, 2003), most literature reports the benefits of incorporating constructivist philosophies in the design of learning packages (e.g. Bruner, 2001), including online learning settings (Palloff & Pratt, 1999). Constructivist environments have been demonstrated to be more effective than traditional settings (Johnson, 1991; Zemsky & Massey, 2005) because they extend students beyond the information presented to them (Bruner, 2001), lead to enhanced cognitive development (Glassman, 2001) and increase motivation (Slavin, 1990), perception of skill development and solution satisfaction (Benbunan-Fich, 1997). Nevertheless, it is widely accepted that simply grouping students together and asking them to work collaboratively does not facilitate the development of a constructivist learning environment or community development (Hiltz, 1998). Jonassen et al (1995) state: ‘These environments should emerge from authentic tasks, engage the learners in meaningful, problem based thinking and require negotiation of meaning and reflection on what has been learned’. Clearly the development of this environment requires purposeful action on behalf of the instructor (Oliver & Herrington, 250 Australasian Journal of Educational Technology, 2007, 23(2) 2001), that may be guided by the nine discrete characteristics of situated learning environments identified by Herrington and Oliver (1995), one of which is the collaborative construction of knowledge. Oliver and Herrington (1995, p. 181) indicate that including collaboration in the design of an educational package, where students are required to engage in higher order thinking and reflection, affords ‘clear educational advantage’. This view is evident in the works of other scholars who assert that the social phenomenon of community might be put to good use in the support of online learning (Bonk & Wisher, 2000; Hiltz, 1998; Palloff & Pratt, 1999). This assertion is well supported by theories of learning that highlight the importance of social interactions in the construction of knowledge (e.g. Bruner, 2001; Dewey, 1929). Further support is found in the works of scholars who explore the community construct. These scholars posit that community is characterised by a willingness of members to seek new members, involve all participants, and share knowledge and the results of their endeavours (Anderson, Annand, & Wark, 2005). Benefits associated with community membership include an increase in intellectual capital (Stewart, 1997), an increase in social capital including the norms of reciprocity (Putnam, 2000), and the satisfaction obtained through membership (Lott & Lott, 1965). It has long been suggested that sense of community is characterised by a phenomenon of the whole being greater than the sum of its parts (Hawley, 1950). These characteristics afford members clear advantage over non members, but it remains unclear in what ways these characteristics might be purposefully developed in online settings (Bonk & Wisher, 2000). It is clear however that the decision to join some communities and not others rests with the will of the individual (Tönnies, 1955). Tönnies (1955) suggested that will falls into two categories, natural will which is associated with the temperament, character and intellectual attitude of the individual, and rational will which is associated with rational decision making. It has been demonstrated that, even in situations where indifference or antipathy are the norm, individuals have exercised rational will to form a community with purpose as the binding factor (Tönnies, 1955). The presence of natural will might explain why some students seek to form learning communities with little intervention from the instructor, whereas the existence of rational will suggests that it might be possible to encourage the formation of a community where students would not normally choose to do so. Clarkson and Brook 251 Aim Using the principles identified in the literature above, the authors sought to integrate the community development construct into an existing unit. The overall purpose was to capitalise on top down principles of institutional graduate attributes policies, and more organic and constructivist principles of collaborative construction of knowledge, authentic activities and reflective practices. In particular it sought to research two topics: firstly, student responses to authentic and collaborative learning activities and peer generated marking keys and secondly, the value of linking an institution’s policies and its learning settings. The research in this paper is part of an ongoing case study into the design of authentic learning environments in an online setting. It was undertaken by the authors, one of whom coordinated and lectured in the unit, the other was an educational settings designer. An earlier phase of the study that focused on incorporating authentic assessment activities into unit design concluded that students perceived disconnectedness between those activities of the lived in world and their assessment. The students felt that the authentic assessment approach to unit design did not enhance their learning experience (Clarkson & Brook, 2003). In an attempt to redress the weaknesses in unit design identified by students and to meet system expectations, the authors sought to revise the assessment activities, ensure stronger links with the institution’s teaching and learning policies, provide stronger links with the lived in world, and promote community development. The results of that earlier redesign phase suggested that students engaged at a commendable level of commitment in group activities and that the increased focus on cognitively authentic tasks was effective in connecting assessment with the activities of the lived in world (Clarkson & Brook, 2004). Those findings led us to consider a specific aim now addressed in this research, namely whether institution mandated generic skills could be embedded in unit design in a way that did not detract from the intent of the unit. The focus on student satisfaction is therefore an important element of this research. Setting The setting describes the particular unit under study and how it fitted into the students’ courses. It details the unit design, the unit processes, the unit assessments and the community development strategies used. The authors were researchers and practitioners, one the unit coordinator and the other taking the role of consultant instructional designer. The unit under study was called Publishing on the Web and was both a first year and fourth year (postgraduate) unit. It was taken by a total of 86 252 Australasian Journal of Educational Technology, 2007, 23(2) students, 10 of whom were postgraduate. It was an undergraduate unit in the Bachelor of Communications degree (B.Comms), taken mainly by students in the Interactive Multimedia (IMM) stream. IMM is one of eight streams, This unit (as IMM1122) is also as an option by a wide range of students from the other seven streams (e.g. Journalism, Mass Communications). It was also available as a fourth year unit for students doing various postgraduate studies (as IMM4122) whose assessment structures were more sophisticated but entirely consistent with the undergraduates’. The unit ran in face to face mode over a 13 week period with all resources and most of the assessment activities being executed online. The unit design This version of the unit had three key purposes. It aimed primarily to teach students about the web and the underlying principles of web page and web site design and construction. The approach used a text editor to show how web pages are constructed as plain text using XHTML and associated underlying constructs. Its second purpose was to provide an authentic setting for students to practice collaboration, research and problem solving. Finally, added this particular semester, was the intention to foreground some of the generic skills policy of the university. At the time the institution had ten attributes divided into two groups named ‘core’ and ‘generic’ attributes. The four ‘core’ attributes were: 1. Enterprise, initiative and creativity 2. Professional knowledge 3. Service 4. Workplace experience or applied competencies. The remaining six ‘generic’ graduate attributes were: Awareness of political, social and ethical issues; Communication; Internationalisation/ cross cultural awareness; Problem solving/ decision making; Teamwork; and Use of technology/ information literacy (ECU, 2004). Instructors were asked by the university to embed, as much as practical, these graduate attributes in the design and implementation of each unit of study developed. We judged it appropriate to allow students at the first year level to identify an attribute or two and show evidence of their achievement. The postgraduates were asked to address two or more attributes. The unit processes Students were expected to participate in four sets of activities mediated by their instructors (the lecturer and three tutors): Clarkson and Brook 253 • weekly lectures by the lecturer – also available online (1 hour) • weekly workshops with 20-25 students and a tutor, where students were constantly online in a computer lab (2 hours) • weekly group research activities, starting week 3, to which students post solutions online and • two individual major, web-centric assignments. The weekly group research activities are described in more detail below. The assignments and the online work were assessable, and interrelated. For example the students’ weekly research generated one section of the marking key – explained below – for their assignments (down from two sections, in the previous semester). The weekly research took place in groups completed in students’ own time, but they were allocated 15 minutes at the start of each workshop for group planning purposes to show that their collaboration was sanctioned and to allow their tutor to support the building of a small community of learners with a common purpose. Each week from week 3, student groups were presented with an ‘ill- defined’ or open ended problem whose scope and complexity needed to be clarified before it could be solved reasonably; for example: ‘What is the best browser for accessibility?’ They were usually called ‘woolly problems’. The rest of the 10 weekly problems fell into two groups: a) general web related issues like e-crime, good page design, graphics formats, and so on; and b) building a part of the marking key for the next assessment. The problems were designed to engage students in authentic activities reflecting a quality assurance process often found in industry practice. Each problem utilised the online tools of URL posting (an asynchronous process) and problem solving (a synchronous one) as well as Forums for feedback purposes, which were easily accessible once logged on (see Figure 1). The marking keys were treated slightly differently. Although they represented woolly problems, they also possessed pointed authenticity, as they became part of the key by which students were marked. Students were told that, based on their votes in the lecture the following week, the teaching staff would choose the ‘best’ one. This chosen key would then be incorporated into each published assignment marking key. Students were shown example template structures based on comparative scales (e.g. 1 to 5 ratings for sub-items) but were free to design a marking key that suited them and their perceived needs as clients of the marking process. In lectures students were given five strategies and related structured advice on ways to address and solve woolly problems in a group setting. For example they were advised to clarify and scope the problem (as the old saw posits, ‘a problem well-defined is a problem half solved’). Individually 254 Australasian Journal of Educational Technology, 2007, 23(2) they did their research and then posted URLs of their results online for others to use. Next in their weekly problems, students shared research and took turns submitting their group’s amalgamated solution to the weekly problem (300 words maximum). Each week they rostered a team member to act as team leader; this person received their emailed research to use towards the team solution. After submitting their solution, the leader was then required to view, reflect on, give feedback on and mark the solutions posted by three other randomly selected teams. They gave a score from 0 to 9 for this. Tutors then provided teams with an online assessment and comment on the completed work, and students had the opportunity to submit further feedback, make comments and suggestions to the Forums. The role of tutor was to provide expert support through scaffolding that was intended to reduce over time. Similarly, reviews of solutions were conducted in lectures each week with decreasing levels of scaffolded support. The final assignment – a prototype e-portfolio – was intended to be extensible through the students’ undergraduate careers until they graduated and sought work in the field. This was described as providing an authentic contribution to their professional careers. This portfolio required students to incorporate evidence that they had attained one or more of the ten graduate attributes identified above, on the grounds that employers would understandably be interested to see that they had attained desirable graduate attributes in their degree. A rich set of online educational tools was available to the unit designer using a custom shareware LMS tool named scamSyte that included blogs, p o r t f o l i o s , s t u d e n t c o n t r a c t s , a n d o n l i n e t e s t i n g m o d u l e s [http://www.scam.ecu.edu.au/]. In this case the modules Schedule, Calendar, Messages, URL posting, Problem Solving and Forums were made available to students (Fig 1). Figure 1: Top row of student’s view of the unit website after login. In this unit the student has six modules available, and in this instance has chosen the URL Posting tool The student sample A total of 76 undergraduate and 10 postgraduate students completed the unit that was the focus of this particular study. For some first year students, this was their first experience in accessing learning resources and Clarkson and Brook 255 engaging online in a collaborative activity. Many of the fourth year students had more experience in such settings. In week 2 workshops all students were introduced to a ‘Groups’ tool used by most of the scamSyte tools. They then chose peers and allocated themselves into groups of four in readiness for the weekly research starting week 3. Community development strategies The selection of community development strategies was guided by the work of contemporary scholars (Hill & Raven, 2000; Kim, 2000; Palloff & Pratt, 1999) and the Learning Community Development Model (LCDM) proposed by Brook & Oliver (2003). Specifically, instructor actions were guided by the process component of LCDM that comprises four elements: 1. Establishing a reason and context for communication: The instructor took intentional action to influence the will of individual students to seek community membership. It was accepted that some of the students would seek community membership as a consequence of a predisposition to do so. These students were further encouraged through the instructor highlighting the availability of unit related discussion boards, and possible uses for these boards. It was also accepted that some students would have a predisposition not to seek community membership. We attempted to influence these students to exercise rational will to seek membership by using authentic activities and embedding the product of their collaborative endeavours in the assessment schedule. Students were not graded on their extent or quality of their collaboration, but on the value of the product resulting from collaborative activity. 2. Enabling communication: Communication tools available to the students included timetabled meetings and asynchronous discussion boards (called Forums in this site). The blended nature of this learning setting also enabled students to communicate via face to face and telephone contact out of class time. A further strategy to enable communication included a schedule for the completion of collaborative products. This schedule was made clear to students and aligned with the assessment schedule. 3. Supporting communication: Students were made aware of the nuances of communicating in text and were provided with a code of conduct to govern their behaviours. In addition, the instructor and tutors provided technical support and encouraged peer support to minimise student difficulties. 4. Moderating communication: The instructor took a role in moderating the discussion board activities. He took intentional action to weave student comments or to promote further discussion. Interestingly, the tutors were 256 Australasian Journal of Educational Technology, 2007, 23(2) not initially asked to be moderators but active participants and engage when they believed it necessary; the role of tutor as moderator was not planned but developed as keener tutors participated. Methodology The study sought to follow Stoke’s’ principle of use inspired basic research (Stokes, 1997) by structuring an inquiry focused on the application of findings and a contribution to the growing theoretical knowledge base. This approach has been adopted to address the limitations of standard experimental design in online environments (Hiltz, 1990) and to avoid the debate over basic versus applied research (Reeves, 2000). The quest for both fundamental understanding and application of findings have been guiding factors in the selection of the research paradigm and also the methodology (Patton, 1990; Stokes, 1997). Largely because qualitative and quantitative paradigms are not mutually exclusive (Patton, 1990), we sought data in this study which would contribute to answering the research questions and our analysis, without feeling the need to differentiate its category. To ensure a direct link between the goals of both researcher/ practitioners the study was grounded in the actions of practitioners and their students. This was facilitated through a case study approach (Burns, 1996) allowing an in depth and focused study of community development (Willig, 2001), and a subjective approach to grounded theory (Merriam, 1998) allowing the study to take place in situ (Strauss, 1987). Whereas the study was guided by what took place, further insight was gained from what did not happen. which was guided by what ought to happen (Patton, 1990). A number of sources of data were used to identify student attitudes and performance over the unit. Formal data sources included the final week questionnaire (see Table 1), and a focus group run by a colleague after the last lecture of the unit. Eight volunteers were asked to provide rationales for answers to the questionnaire, general comments and pass judgements on the unit aims as they understood them. The data was recorded, transcribed by a research assistant and then summarised by the colleague. The remaining sources were available data, disguised where necessary to preserve anonymity. They included the focus group interview; student comments on the Forums and other submitted materials; instructor observations; student usage of tools, particularly the online ones; logon records; and formal university evaluations. Potential incongruence between what the interviewee says and what actually happens will be explored through an observational data collection strategy (Becker & Blanch, 1970). Observations of the learning setting will Clarkson and Brook 257 be made to gauge the extent that students demonstrate the graduate attributes described in the system policy document. Observations will follow a structured approach proposed by Kiddler (1981). 1. What should be observed? 2. How should observations be recorded? 3. What procedures should be used to try to assure the accuracy of the observations? 4. What relationship should exist between the observer and the observed, and how should such a relationship be established. The observation schedule provides for the opportunity to measure student practices during the learning experience. Observations will be facilitated through online technologies. This approach ensures that student activity is recorded as a semi-permanent record that can be analysed and referred to over time. This feature provides the observer the opportunity to refer back to the practice as if they were referring to the actual event (Merriam, 1998). Resulting qualitative data sets will be analysed using a constant comparative approach (Patton, 1990). Qualitative data will be coded according to emergent themes. Themes will be constantly compared with emergent categories to establish a best fit with the data set. Quantitative data collected through the survey will be analysed using descriptive statistics in accordance with the limitations of a relatively small sample size. This research sought confirmation that conclusions were meaningful by standard qualitative means; for example the variety of data sources provided increased confidence in the findings through triangulation (Mills, 2003). Findings The findings are presented in order from a student perspective, an instructor perspective and then using the formal data sources available, namely the student logon rates (Figure 2), the week 13 questionnaire (Table 1) and formal university evaluations for the unit, which became available early in the following semester. The combination of qualitative and quantitative data provides evidence that the students’ perspectives, participation and performance all contribute to the idea that students appeared to find the online activities challenging, and that they felt satisfaction with the learning experience which integrated generic skills development into their course. Student observations The lecturer and instructors encouraged regular use of the available tools over the semester, to promote and maintain students’ engagement. 258 Australasian Journal of Educational Technology, 2007, 23(2) However, the most influential factors encouraging student participation appeared to stem from a sense of commitment to each other and an unwillingness to let the others down or negatively affect the grades allocated to the completion of group tasks. These behaviours are often associated with a strong sense of community. Feedback from students in the focus group clarified this. For example, Student A, a third year, commented: The weekly structure helped; it makes you do it every week, it makes you do it as a team, it let’s you get feedback and then you read the other people’s and get their… take on it as well. For others, collaboration was not easy, with some students finding it difficult to allocate the required time to engage in collaborative activity – a factor that has been shown to impede community development. Student B (4th year) argued that the team provided encouragement through the pressure of not wanting to let others down: You just had to do it and you didn’t want to let your team down… Some weeks I actually just thought ‘I can’t afford the time this week’ but because you’re working in a team effort you just gotta do it, yeah. Student C, a first year, acknowledged that the group work was more compelling than the individual work, at least because of the structure of the marking scheme, and this guided some students’ use of time: … I realise that I focussed … on the [weekly group research] questions rather than on the [individual class] html activities, which weren’t marked. Groups were occasionally disrupted for two reasons, either when a student left the unit and those remaining needing to re-group, or when a student did not collaborate successfully as seen in this comment by student D, a second year: We had the problem of ’Larry’ [who] was working full time. He did football or something and it was really difficult to get together. … like a day later waiting for someone to email and no responses… Although occasionally students made negative comments on the challenges of collaboration or the disruptive nature of students leaving the unit, there seems no simple solution to these issues. The marking key activity (see the Week 13 questionnaire as well) seemed to have contributed to the relevance of the unit. This feedback suggests the students did not find the unit trivial and that it required significant collaboration. Remembering that nearly all the weekly research activities were completed online (requiring logon), these student observations were the first evidence of some online community development, based on activities fostering the generic skills of the title of this paper. Clarkson and Brook 259 Instructor observations Initially all the tutors were worried that the incorporation of University mandated graduate attributes in assessment activities was going to be extremely unpopular with students and seen as a policy decision imposed from above. It was decided that to counteract this, a two step approach in the assignment should be implemented, by asking the students to: ‘i) identify one or more attributes you want; and then ii) provide evidence that you have met it/them at least once (e.g. employer’s reference, a relevant email, a certificate, etc.)’. This process ensured a direct link between the assessment activity and the lived in world of the students by embedding the assessment activity in their professional portfolios. This process turned out to be easy to explain and demonstrate, and seemed acceptable to the students. The feedback of all instructors at assignment marking time was that many students demonstrated a sound understanding of graduate attributes and integrated them into their e-portfolios. Although this was an individual process, there were significant entries on the Forums about these attributes, implying that there had been useful discussion on the topic for many students. The problem solving activity over the 10 weeks produced some rather interesting results. First, there was a very gradual improvement in average scores for most tutor groups over the semester, suggesting that either the tutors were marking more easily or the students were getting better. The instructors all expressed the view that the students had developed better approaches to woolly problems over the semester. Secondly, the range of peer marks allocated to their three random groups increased over the semester – as measured by the slope of the trendline of peer marks each week gradually becoming steeper – suggesting that they were becoming more discriminating in their judgements about what were high and low quality responses. Thirdly, there were a few negative comments from some outspoken students during the middle period of the unit, which became supportive comments after the unit was completed (the week 13 questionnaire, below, could be seen as confirming this). Finally, predictably, the marking key component was handled much better by the 4th year students and there was never dissention when a solution provided by a 4th year was chosen. Overall, the students gave supportive feedback to their instructors via emails, the Forums and verbal comment, and thus arguably contributed at a very satisfactory level to the processes of the unit. Some students commented on the breadth of activities and inferred that the workload was higher than other units. This is hard to prove but since the instructors also teach in other units and know the expectations of other lecturers, then on 260 Australasian Journal of Educational Technology, 2007, 23(2) balance this seems relatively unlikely; instructors felt that the workload was within the normal range of workloads, but perhaps may have been closer to a 2nd year unit than a 1st year unit. Two students complained very early in the unit about the expectation about collaborating with other students, which seemed rather ingenuous in the current multimedia environment with its increasing emphasis on large projects, teamwork and collaboration. The student feedback was seen to acknowledge that the activities were relevant, largely authentic and contributed to their development of some generic skills. Logon records The students generally logged on over the 13 weeks at a rate shown by the two charts in Figure 2. The two graphs show total logons over the 13-week period and ranged from low to very high. Students logged on to check unit materials, view lectures and workshop notes, post URLs and problem solving solutions, read and write to Forums and read system messages. The most common reason to logon was to contribute to the weekly research activity, which started in week 3, so only 10 weeks of submissions were involved and this is probably the period when most logons occurred. In the mainly 1st year group (upper graph), the average rate of access was 107 times or very roughly 9 times a week. More interesting was the variability, with the lowest 21 and highest 353 (about 3.5 times average). To sum up: no one logged on fewer than 21 times over the semester, and even the lowest group of 5 students, about 7%, logged on between 21 and 29 times to average around twice a week. In the 4th year group of 10 students the average was 221 logons, roughly double the first years. There was even more variability; the lowest logon count was similar to first years at 20 logons total. The highest figure (nearly 1000 logons or about 80 times a week!), was a very keen student who was determined that everyone in that group should perform well and kept logging on to encourage (or more usually chastise!) them. Two of them mentioned frustration with such an approach during the semester; the lecturer offered to intervene but recommended they do so themselves in the interests of their ‘community’. The issue was not so much resolved as reduced after they arranged a small meeting. Since this single figure was so disparate, it seemed reasonable to exclude it from the calculation of the average logon rate, above. The rate of logon is rather high for all students who were simply viewing notes or reviewing a weekly lecture; this rate is better explained by the various online tools including the Forums; the URL posting tool, which required reviewing others URLs; and the problem solving tool which Clarkson and Brook 261 required research, sharing, collaborating, submitting and then assessing others’ online solutions around a weekly activity schedule. At this stage the similarity of the graphs for undergraduate and postgraduates led us to believe that the simplest explanation for the differences was more about scale than anything of deeper significance. Logon Count (U/grad students) 0 200 400 Students Logon Count (4th yr students) 0 200 400 1 2 3 4 5 6 7 8 9 10 Students Figure 2: Total logon count over the semester for all undergraduates (76 students, upper graph) and 4th year (10 students, lower graph) to the website. Each column shows the number of logons for that student ID, sorted by logon count from low to high. Note that the largest 4th year logon of 979 is well off the scale. Overall a significant level of commitment by nearly all students was detected; there was a gradual downward slope to the left on each graph suggestive of diminishing social and time commitment as the level of logon dropped; and we noted that one student in each group seemed to log on far more than most. A study of the characteristics and scores of these ‘extreme’ or over keen students could be instructive in future research. Week 13 questionnaire Students attending the last lecture (around 35 of 76 enrolled was considered acceptable for a unit that relied on authentic assessments and had no formal exam; and not all of them completed the questionnaire nor 262 Australasian Journal of Educational Technology, 2007, 23(2) the formal university evaluation mentioned below) were asked about the learning activities they had experienced. Arguably these were the keener students of the original cohort and may under-represent negative opinions and feedback. The questions (see Table 1) asked about the use of authentic activities including student constructed marking keys and the incorporation of graduate attributes, the problem solving tool, the URL posting tool, and collaboration (teamwork). In particular, questions 1, 2, 5, 6 and 7 were intended to give a measure of their perceived value. Items 3 and 4 were intended to give a measure of their level of enjoyment of some of the tools. A 5-point Likert type scale was used (1=Strongly disagree, 3=Neutral and 5=Strongly agree) which was easily graphed with 3 as the neutral point (see Figs. 3, 4 and 5). Overall, all the ratings were above 3 (averages ranged from 3.3 to 3.9), showing that they could all be regarded as helpful. It is instructive that Q3 about the problem solving tool, although positive, shows that it was the least ‘popular’ question; and yet Q6, about its value in learning, shows it was the most helpful of any of the tools. Obviously something about the tool or its presentation needs refinement. Table 1: Questions and responses to the voluntary and anonymous questionnaire of week 13 (N=28) Question Averagescore (/5) SD 1 Making marking keys was a useful learning activity 3.8 0.8 2 The assessment tasks used largely authentic activities 3.8 0.5 3 I enjoyed using the problem solving tool 3.3 1.0 4 I enjoyed using the URL tool 3.6 0.5 5 The problem solving tool and the given research strategies helped me solve ill-defined problems 3.6 1.0 6 The problem solving tool helped my learning 3.9 0.8 7 Teamwork was a useful and necessary part of this unit 3.7 1.1 8 I would support these approaches in other units 3.5 0.9 Signs of student support for the variety of ‘authentic activities’ are provided in answers to Qs 1, 2, 5, 6 and 7. Of these, the graphs for Qs 1, 2 and 7 are instructive. Note that the data are sorted on responses to Q6. Q1 (Figure 3) shows that the vast majority appreciated the value of constructing their own marking keys. Students 4, 25 and 26 are the only dissenters. Q2 (Figure 4) had the least variability (SD 0.5) and most students chose 4 (i.e. “Agree”). Interestingly, only one student felt strongly enough to score it 5, but no one disagreed with the statement. Finally Q7 (Figure 5) addresses a thorny topic, namely that, for some students, teamwork is perceived as not useful, or helpful, or that they dislike it. Two students, 4 and 12, felt strongly, including student 4 who Clarkson and Brook 263 was also one of very few who was negative about the value of marking keys (Figure 3). Student 12 was neutral about marking keys. Q1: Marking keys 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Figure 3: Q1 responses on student-generated Marking Key construction Q2: Activities authentic 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Figure 4: Q2 responses on whether the assessments were largely authentic Q7:Teamwork useful, necessary 1 2 3 4 5 1 3 5 7 9 11 13 15 17 19 21 23 25 27 Figure 5: Q7 responses on whether teamwork was useful and necessary Although the online problem solving tool may need refinement, for the 28 students who responded, the process seems to have proved adequate as a method for improving skills and encouraging community. Interestingly the fact that only two students complained to an instructor about the teamwork activities during the semester and both of them appeared to turn up at the week 13 lecture weakens the suggestion at the start that the week 13 feedback may be too positive and not representative. Overall the students appear to be agreeing that the online tools were useful and contributed to the development of some of their generic skills, but there was less certainty on the value of teamwork for these students. Taken in conjunction with the student observations earlier, the sense of community seems to be associated with the success of the online problem solving tool. 264 Australasian Journal of Educational Technology, 2007, 23(2) Formal university evaluations Each semester the University asks students to rate the quality of the unit, lecturer and their tutor (Table 2). As an example the lecturer ratings are given for the previous and the current semesters in Table 2. The ratings range from -100 to +100, and the average for all units scored by the school is also provided. At the same time that the assessments have been made more authentic, and the weekly problem solving and marking keys slightly simplified, these formal lecturer ratings improved from the previous semester (middle column) to the newer one (last column). With case studies such as this the intention is not to make broad conclusions but to understand what is going on (Gay, Mills & Airasian, 2006). Table 2: Comparison of the student evaluations (which range for -100 to +100) over the two semesters described in this research on the unit. Lecturer assessment scores (-100 to +100) Prev. sem (N=34) Study sem (N=17) Mean score 49 79 Mean score for school units 47 35 The fact that the average data across all students and many tens of courses in the school was also rather changeable (dropping from 47 to 35) was a concern. Nevertheless there was a simple trend, namely that at the same time that the school mean fell, the mean for this unit rose. Arguably it is the direction of the change of score, from just below 50 to just below 80 out of 100, rather than any raw numbers which is important in this table, suggesting that the unit content and presentation has improved. Such data only indicates that students appear happy with the unit overall; but when taken in conjunction with the previous data from students, instructors and their activities it would seem that a sense of online community was evident. Furthermore, the critical research aim, to investigate the inclusion of top down generic skills requirements that may be in this unit, has not proved problematic, arguably insulated by the focus on facets like community and authentic activities. It seems also that students acknowledged the value of the online tools in contributing towards the development of some of their generic skills. Importantly these skills were useful and relevant to the unit itself. Conclusion This paper reports on changes to a tertiary unit that had a joint focus upon enhancing the authenticity of the assessments, and incorporating one of the University’s newer policy directives, namely graduate attributes. The data suggest that students engaged at a commendable level with the online Clarkson and Brook 265 group activities in this unit, based on the written and spoken feedback presented and the logon rates. The nature of student comments regarding collaborative work, the factors that motivated their participation and even the high logon rates, suggests a high level of commitment to group work and each other. These characteristics are often associated with a strong sense of community. The increasing focus on cognitively authentic tasks seems to have been effective, based on the good level of feedback received and questionnaire analysis. The weekly activities appear to have impacted positively on the learning experience with students expressing quite strongly that they had learned useful skills in the unit, including the generic skills sought. In hindsight, it transpires that the task of integrating the attributes was easier than expected, as the activities undertaken (student generated marking keys, URL posting online, authentic weekly problems using an online tool, multiple collaboration opportunities, graduate attributes within a prototype e-portfolio assignment) were able to integrate into one another. One argument could be that this is because such attributes are an easy policy directive to implement, but it is our view that the online setting described above materially helped – that asynchronous and synchronous tools helped achieve synchronicity from the students’ perspective. In the end we identified that although graduate attributes may not have appealed to all students, embedding them into a unit’s assessment was workable. It assisted in making links between the learning materials and the lived in world of the students, because of the connection with employability at the end of such a course. We cannot make conclusions about whether students showed improved performance on their personal generic skills, as this was not the focus of the research and insufficient data was collected towards that aim. Nevertheless, as universities increasingly ask for evidence that generic skills like these are outcomes for all students’ tertiary education, the relative ease with which they were incorporated into a first year unit suggests that they may be just as easily integrated into second and third year units for older and more mature students. It appears that the tools described here - activities supporting community development online - contributed to the successful embedding of graduate attributes such as the capacity to work in team settings. As a result, due to the use of the asynchronous and synchronous online tools mentioned in this study, a better synergy was evident between the goals of the institution and those of the lecturer as seen by the performance of the students in their classroom. 266 Australasian Journal of Educational Technology, 2007, 23(2) References Anderson, T., Annand, D. & Wark, N. (2005). The search for learning community in learner paced distance education: Or, 'Having your cake and eating it, too!' Australasian Journal of Educational Technology, 21, 222-241. http://www.ascilite.org.au/ajet/ajet21/anderson.html Becker, H. & Blanch, G. (1970). Participant observation and interviewing: A comparison. In W. J. Filstead (Ed.), Qualitative Methodology. Chicago: Markham. Benbunan-Fich, R. (1997). Effects of computer-mediated communication systems on learning, performance and satisfaction: A comparison of groups and individuals solving ethical case scenarios. Unpublished Dissertation, Rutgers University - NJIT. Bonk, C. J. & Wisher, R. A. (2000). Applying collaborative and e-learning tools to military distance learning: A research framework. [viewed 2 July 2002] http://www.publicationshare.com/docs/Dist.Learn(Wisher).pdf Brook, C. & Oliver, R. (2003). Online learning communities: Investigating a design framework. Australian Journal of Educational Technology, 19(2), 139-160. http://www.ascilite.org.au/ajet/ajet19/brook.html Bruner, J. (2001). Constructivist theory. [viewed 10 May 2001] http://tip.psychology.org/bruner.html Burns, R. B. (1996). Introduction to research methods. South Melbourne, Australia: Addison Wesley Longman Australia Pty. Limited. Clarkson, B. & Brook, C. (2003). Whose standards?: Asking students to generate the marking key in a multimedia unit. Paper presented at the Ed-Media 2003 - World Conference on Educational Multimedia, Hypermedia & Telecommunications, Hawaii. Clarkson, B. & Brook, C. (2004). A Focus on Graduate Attributes: Aligning policy, practice and learning in an undergraduate multimedia subject. Paper presented at the Ed-Media 2004 - World Conference on Educational Multimedia, Hypermedia & Telecommunications, Lugano. Dewey, J. (1929). The sources of a science of education. New York: Liveright. ECU. (2004). Graduate Attributes @ ECU. [viewed 12 May 2006] http://www.ecu.edu.au/LDS/rd/units/graduate_attributes.html Gay, L. R., Mills, G. E. & Airasian, P. W. (2006). Educational research: Competencies for analysis and applications. Upper Saddle River, NJ: Merrill/Prentice Hall. Glassman, M. (2001). Dewey and Vygotsky: Society experience and inquiry in education practice. Educational Researcher, 30(4), 3-14. Hawley, A. H. (1950). Human ecology: A theory of community structure. New York: The Ronald Press Company. Clarkson and Brook 267 Herrington, J. & Oliver, R. (1995). Critical characteristics of situated learning: Implications for the instructional design of multimedia. Proceedings ASCILITE Conference, University of Melbourne. http://www.ascilite.org.au/conferences/ melbourne95/smtu/papers/herrington.pdf Hill, J. R., & Raven, A. (2000). Online learning communities: If you build them, will they stay? ITFORUM PAPER #46 [viewed March 2001, verified 28 Apr 2007] http://it.coe.uga.edu/itforum/paper46/paper46.htm Hiltz, S. R. (1990). Evaluating the virtual classroom. In L. M. Harasim (Ed.), Online education: Perspectives on a new environment (pp. 133-183). New York: Praeger. Hiltz, S. R. (1998). Collaborative learning in asynchronous learning environments: Building learning communities. Paper presented at the WebNet 98 World Conference of the WWW, Internet and Intranet Proceedings, Orlando, Florida. Johnson, W. D. (1991). Student-student interaction: The neglected variable in education. Educational Research, 10(1), 5-10. Jonassen, D., Davidson, M., Collins, M., Campbell, J. & Haag, B. B. (1995). Constructivism and computer mediated communication in distance education. The American Journal of Distance Education, 9(2), 7-26. Kiddler, L. H. (1981). Sellitz, Wrightsman & Cook's research methods in social relations (4th ed.). Austin, Texas: Holt, Rinehart and Winston. Kim, A. J. (2000). Community building on the web. Berkeley: Peachpit Press. Lott, A. J. & Lott, B. E. (1965). Group cohesiveness as interpersonal attraction: A review of relationships with antecedents and variables. Psychological Bulletin, 64, 259-309. Luca, J. & Oliver, R. (2003). A framework to promote learning and generic skills. In P. Kommers & G. Richards (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1588-1595). Chesapeake, VA: AACE. Merriam, B. S. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass. Mills, G. E. (2003). Action research: a guide for the teacher researcher (2nd ed. ed.). Upper Saddle River, NJ: Merrill/Prentice Hall. Oliver, R. & Herrington, J. (2001). Teaching and learning online: A beginners guide to e- learning and e-teaching in higher education. Perth: Centre for Research in Information Technology and Communications, Edith Cowan University. Palloff, R. & Pratt, K. (1999). Building learning communities in cyberspace. San Francisco: Josey-Bass Publishers. Patton, M. Q. (1990). Qualitative evaluation and research methods. Newbery Park, California: SAGE Publications, Inc. 268 Australasian Journal of Educational Technology, 2007, 23(2) Putnam, R. D. (2000). Bowling alone. New York: Simon and Schuster. Reeves, T. (1999). The scope and standards of the 'Journal of Interactive Learning Research'. Journal of Interactive Learning Research, 1999 (September), 1-6. http://www.aace.org/pubs/jilr/scope.html Reeves, T. (2000). Enhancing the worth of instructional technology research through design experiments and other development research strategies. Paper presented at the International Perspectives on Instructional Technology Research for the 21st Century, New Orleans, LA, USA. [verified 28 Apr 2007] http://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf Rogers, E. M. (1983). Diffusion of innovations (2 ed.). New York: The Free Press. Slavin, R. E. (1990). Cooperative learning theory: Research and practice. Needham Heights, Massachusetts: Allyn and Bacon. Stewart, T. A. (1997). Intellectual capital. New York: Doubleday/Currency. Stokes, D. E. (1997). Pastuer's quadrant: Basic science and technological innovation. Washington, DC: Brookings institution Press. Strauss, A. L. (1987). Qualitative analysis for social scientists. New York: Cambridge University Press. Tönnies, F. (1955). Community and association (C. P. Loomis, Trans.). London: Routland & Kegan Paul Ltd. Willig, C. (2001). Introducing qualitative research in psychology: Adventures in theory and method. Buckingham: Open University Press. Zemsky, R. & Massey, W. F. (2005). Stalled: E-learning as thwarted innovation. In A. A. Carr-Chellman (Ed.), Global perspectives on e-learning: Rhetoric and reality (pp. 241-255). Thousand Oaks, CA 91320: Sage Publications, Inc. Dr Barnard Clarkson, Multimedia Coordinator School of Communications & Contemporary Arts Mr Christopher Brook, Post Doctoral Scholar Faculty of Education & Arts Mount Lawley Campus, Edith Cowan University 2 Bradford Street, Mt Lawley WA 6050, Australia Email: b.clarkson@ecu.edu.au, c.brook@ecu.edu.au Web: http://www.scca.ecu.edu.au/