Microsoft Word - byrnes.doc Australasian Journal of Educational Technology 2006, 22(1), 104-125 The prevalence and characteristics of online assessment in Australian universities Rod Byrnes and Allan Ellis Southern Cross University Assessment is one of the key elements of the teaching and learning process. It provides teachers with a means of evaluating the quality of their instruction. Students also use it to drive and direct their learning. Online teaching and learning will continue to become more important to Australian universities in order for them to remain competitive and economically viable. In the online environment, assessment is no less critical than in traditional face to face environments. However, assessment risks being overlooked or at least marginalised in the rush to place course content online. This paper provides a snapshot of the prevalence and characteristics of online assessment in Australian universities during 2004. It highlights useful information regarding the use of online assessment in the university sector and illustrates that overall this crucial area is not being given the attention or resources it requires. Introduction Australian universities need to take online education seriously. Oliver (2001) states, “technology will play a large role in the planning, development and delivery of the curriculum of the contemporary university”. This view is backed up by Erdinc (2002) who notes that online eduction already “plays a major role in the delivery of education.” Oliver (2001) concludes his statement by saying “the challenge for institutions is to make decisions now that will set them on the preferred and appropriate path for the future.” Assessment is one of the key elements of the teaching and learning process. It provides instructors with a means of evaluating the quality of their instruction (Kerka & Wonacott, 2000). Students use it to direct and drive their learning (James, McInnis & Devlin, 2002; Ramsden, 1992, in Kendle & Northcote, 2000). It is also essential in maintaining standards related to certification (Black, 1998). Byrnes and Ellis 105 In the online environment, assessment is no less critical than in traditional face to face environments. Thorpe (1998: 265, in Kendle & Northcote, 2000) explains that “assessment has been recognised as a driver of students’ approaches to study in distance education no less than in campus based settings”. Kendle and Northcote (2000) conclude that “as such, assessment should be one of the first design considerations when preparing an online course, and be seamlessly integrated into the course, not ‘tacked on’ as an afterthought.” Harris, et al. (2002) define online assessment as “an entirely automated process of delivering and marking assessments using Web or Internet resources”. Within the boundaries of this definition, online assessment can be either formative or summative. Formative assessment refers to assessment that is used primarily to provide feedback and further instruction to the student or inform the instructor of the student’s progress (Bull & McKenna, 2004: xiv). In contrast, summative assessment refers to assessment that is primarily used to evaluate a student’s performance and contributes to their final grade (Bull & McKenna, 2004: xiv). Curoe (1999: 26) highlights the importance of formative assessment by emphasising that “providing several opportunities for elaborative rehearsal and frequent practice all support the way we know how learners learn”. One advantage of online assessment is that once set up, students are able to utilise formative assessment resources numerous times without instructor intervention. This leads to better learning outcomes (Buchanan, 2000; Peat, et al., 2004) without the equivalent increase in instructor workload that would be the case with more traditional forms of assessment. The move to online teaching and learning, in order to remain competitive, coupled with the importance of assessment as a critical component of the teaching and learning process, may in and of itself be sufficient justification for the use of online assessment. However online assessment implicitly holds the promise of a number of other substantial benefits. These include the ability to assess larger classes effectively (Booth, et al., 2003: 44; White & Davis, 2000); increase student motivation (Bull & McKenna, 2004); increase feedback (Booth, et al., 2003: 42; Bull & McKenna, 2004: 5); increase flexibility (Australian Flexible Learning Framework Quick Guides, 2002; Booth, et al., 2003: 42); increase the objectivity and consistency of marking (Bull & McKenna, 2004: 6-7); and increase administrative efficiency (Bull & McKenna, 2004: 7-8; Byrnes, 1994: 2). Online assessment also has the potential to offer a greater range of assessment methods, some but not all of which are available in a traditional face to face settings. These include adaptive testing (Booth, et al., 2003: 25; Bull & McKenna, 2004: 76-77), online discussion forums (Booth, et. al, 2003: 32-33), online portfolios (Booth, et al., 2003: 34), role plays (Australian 106 Australasian Journal of Educational Technology, 2006, 22(1) Flexible Learning Framework Quick Guides, 2002), authentic simulations (Booth, et al., 2003: 33, 35, 66, 79) and games (Bull & McKenna, 2004: 100- 101). In terms of quality, online assessment should not differ from traditional assessment approaches. Booth, et al. (2003: 8) notes that the “principles of quality assessment are the same for online as they are in any form of delivery. These include … the need for assessment to be valid, reliable, fair and flexible”. Survey of Australian universities Little research exists on the prevalence and nature of online assessment technology in Australian universities. Gallagher comments on this lack of information with regard to e-learning in general: The extent of use of e-learning in universities in Australia – that is the number of units or courses taught using e-learning methods – is also difficult to determine. Although there are occasional reports which proclaim significant advances in the use of ICT in education … it is often difficult to track down the actual statistics, the methodology or definitions used for their collection. (Gallagher, 2001: 6) One study that is available was conducted by the Department of Education, Science and Training (DEST) in 2002. They surveyed 40 of the 43 (then) universities in Australia as to the use of various types of online education delivery. This included the number of courses that were fully online (“university award courses in which all subjects are delivered and all interactions between staff and students are conducted via the Internet”), the number of units (“subjects or course components”) that were online (“at least some of the content is delivered and/or some of the interaction is conducted via the Internet”), and online services (“university services and information, which students can access via the Internet”). The DEST study focused primarily on online delivery and did not specifically consider issues of assessment. The study found that 207 courses were offered fully online, the majority of these (90%) being at the postgraduate level. There was a marked difference between disciplines based on the presence of fully online courses. With regard to the use of online units, the study found that all universities used the Internet to some extent for teaching and learning delivery. Overall, 54 per cent of units had some online content. Only 1.4 per cent of units were available fully online. Once again, the majority of these were postgraduate units, and there was a marked difference between disciplines. Byrnes and Ellis 107 Survey of Australian academics As a consequence of the limited amount of existing research on this topic, and to update and overcome shortcomings in the DEST study which was two years old and didn’t specifically address online assessment, it was decided there was a valid need to conduct a more up to date survey of Australian academics that specifically addressed the issue of online assessment. The survey was developed to provide answers to the following questions: 1. How prevalent is the use of online assessment? 2. Does the use of online assessment vary with university size and age? 3. What types of online assessment are in use? 4. How prevalent are online summative final exams? 5. Is the use of online assessment related to discipline area? 6. What is the perceived difficulty of supporting mixed modes of delivery? 7. Should all units be provided fully online? 8. What is the impact of available Learning Management Systems (LMS) on online assessment? 9. What is the impact of available support groups on online assessment? 10. What other institutional factors are involved in the use of online assessment? This may include factors related to institutional policy and the university’s commitment to online assessment. The survey was conducted during 2004 and consisted of 19 questions. A pilot version of the survey was tested on several academics from Southern Cross University in order to evaluate fitness for purpose and eliminate as much ambiguity as possible. The survey questions were grouped as follows: • Two demographic questions regarding the university and discipline of the participant. These were used to determine whether the university (and hence factors such as university size, age or specific policies) or discipline area had an effect on the use of online assessment. • Five questions relating to units currently being taught. These were used to gauge the amount of online coursework and online assessment being used in units. • Six questions concerning the types of online assessment in use. These were used to assess the use of a final summative exam in both online and non-online units, and to determine the types of online assessments in use. • One question relating to mixed (dual) delivery modes (i.e. face to face and online). This question was used to determine the perceived difficulty of supporting more than one mode of delivery – whether is was more difficult or whether economies of scale made it easier. 108 Australasian Journal of Educational Technology, 2006, 22(1) • One question relating to attitudes towards fully online units. This was used to determine whether a person’s attitude towards online assessment affected their use of it, or whether the factors affecting the use of online assessment should be sought elsewhere. • One question concerning the available LMSs provided by the institution. This was used to determine whether particular LMS implementations may have facilitated or hindered the use of online assessment. • One question concerning available support groups. This question was used in an attempt to determine whether the type of support for online assessment had an effect on the amount of online assessment in use. • One question for general comments. • One question requesting optional contact details for following up certain points if required. The required size of the sample was determined by first estimating the size of the population. Only the population of associate lecturers, lecturers and senior lecturers were targeted. This was because the survey was designed to elicit responses from those academics most likely to be at “the coal face” of course design and delivery, representing the core teachers who make the bulk of decisions relating to the use of online assessment. The Australian Bureau of Statistics (ABS) Web site (2004) revealed that in 1992 there were 25,245 academic staff who were lecturers, senior lecturers or above senior lecturers (although these numbers do not correspond directly to the surveyed population, the figures are representative of the numbers of academics in Australian institutions, and they were considered suitable given that there was substantial overlap between the ABS data and the surveyed population and that no other data was available). In 2002 this number was 28,013. This represents an increase of about 1.5 per cent per year. Extrapolating this growth rate from 2002 to 2004 there would be roughly 28,834 academic staff at the level of lecturer or above. For a sample from a population of this size to be statistically representative at the 95% confidence level with a precision of ±5 per cent and maximum variability assumed, an algorithm formulated by Cochran (1963: 75) indicated that 380 responses would be required. Discussions with colleagues suggested a response rate of 20-30 per cent was typical for mail based surveys. As no data was available for similar email based surveys, a 20 per cent response rate was assumed and with rounding it was decided to initially target 2,000 Australian academics. With a 20 per cent response rate, this would have resulted in 400 responses, which was statistically adequate for this study. It was intended that a second mailout be conducted if the number of responses from the first Byrnes and Ellis 109 mailout was less than expected or not evenly distributed. As will be discussed further in the following section, a second mailout was required that targeted an additional 4,441 academic staff members in order to reach the required sample size. The email addresses were harvested in a pseudo-random manner from the online staff directories of Australian universities and other higher degree granting institutions. In all, participation was invited from 34 of the 44 Australian higher degree granting institutions as not all institutions provided email addresses in their publicly accessible staff directory. The addresses were screened to ensure that only associate lecturers, lecturers, or senior lecturers were selected. Table 1 shows the distribution of universities that were sampled, based on the type of university. Universities were grouped based on the period in which each was founded. The groups include the early sandstone universities (1850-1911), the post-war universities (1946-1951), Menzies era universities (1961-1974), Dawkins reform universities (1980s), and other universities and higher degree granting institutions that do not fit into the previous groups. Table 1: Types of universities sampled Group Number of universities in group Number of universities targeted Percentage of universities targeted Sandstone 6 4 67% Post-war 4 3 75% Menzies era 10 8 80% Dawkins reform 13 12 92% Other 11 7 64% Table 1 shows that although not all universities were sampled, the distribution of universities sampled was largely representative of the overall population. It should be noted that the number of addresses obtained for each sampled institution was not the same. This was due to two reasons. Firstly, not all institutions had the same level of functionality in their online staff directories, as the ability to automatically search for staff based on position was not possible for some institutions, and this information had to be collected by individually examining each staff members’ details. Therefore the level of difficultly involved in generating a suitable number of addresses for the institution affected the number that were eventually selected. This was not considered to bias the sample as all institutions were well represented, and staff members within those institutions were chosen 110 Australasian Journal of Educational Technology, 2006, 22(1) in a pseudo-random manner. Secondly, smaller institutions had fewer academic staff members, and in some cases all relevant academic staff members at those institutions were invited to participate. In all, 28 of the 34 surveyed institutions had over 100 invitations sent, with the remainder averaging over 50 invitations each. Survey sample Of the initial 2,000 survey invitations sent, 57 responses were received. Two weeks after the first mailout, a secondary mailout to the same invitees was conducted. This secondary mailout more than doubled the number of responses received to 118, giving an overall response rate of 5.9 per cent. The total number of responses required for this research was 380; therefore an additional 262 responses were required. Assuming the same response rate of 5.9 per cent would be received for the second attempt, another 4,441 email addresses were gathered using the same method as for the first attempt, and the survey was extended. Initially 132 more responses were received. As with the first attempt, a follow up email was sent after two weeks and this also more than doubled the number of responses to 273. Overall, 391 responses were received, which was more than the 380 required for a statistically representative sample. One concern with the sample was that it would be skewed in favour of online assessment due to the probability that invitees would be more likely to respond if they had an interest in online assessment. Surprisingly the results showed that approximately half of the respondents used no online assessment in their units. Although it’s not possible from this data to determine what percentage of academics in the overall population used no online assessment, it does indicate that the bias is not as severe as it may have been. All results, however, should be interpreted cautiously. It should be noted that not every respondent answered every question, and some of the questions were not relevant to every respondent. Therefore the following analyses of those questions may report a sample size less than 391. Table 2 shows the distribution of higher degree granting institutions from which the responses came (n=34). Responses came from all 34 of the Australian universities and other higher degree granting institutions that were targeted. Byrnes and Ellis 111 Table 2: Distribution of responses from Australian higher degree granting institutions (n=34) University No. of responses University No. of responses Australian Catholic Uni. 16 Swinburne University 10 Australian Maritime College 4 University of Ballarat 11 Avondale 5 University of Canberra 11 Central Queensland Uni. 1 University of Melbourne 11 Charles Darwin University 13 University of Newcastle 15 Charles Sturt University 17 University of New England 18 Christian Heritage College 1 Uni. of New South Wales 7 Curtin University 18 University of Queensland 11 Deakin University 15 University of South Australia 17 Edith Cowan University 19 Uni. of Southern Queensland 1 Flinders University 10 Uni. of the Sunshine Coast 5 Griffith University 18 University of Sydney 16 James Cook University 11 Uni. of Technology Sydney 12 Macquarie University 15 Uni. of Western Australia 14 Murdoch University 13 Uni. of Western Sydney 14 Queensland University of Technology 8 University of Wollongong 11 Southern Cross University 12 Victoria University 11 No demographic data was collected about the respondents’ personal identity. Information was collected about the respondents’ discipline, and although discipline names varied slightly from university to university, the responses to this question in the survey can be grouped as shown in Table 3. Table 3: Distribution of responses based on discipline area (n = 391) Discipline area No. of responses Health Sciences and Psychology 60 Education and Communication 58 Business, Maths and IT 57 Social Science and Humanities 38 Economics and Accounting 24 Physical, Chemical and Environmental Science 22 Marketing and Management 20 Biological Science 19 Arts 18 Engineering 18 Law 18 Nursing 16 Other (Miscellaneous, Unspecified) 16 Hospitality and Tourism 7 Total 391 112 Australasian Journal of Educational Technology, 2006, 22(1) Survey results The results of the analysis of surveyed data in terms of the ten questions asked previously are presented below. Question 1: How prevalent is the use of online assessment? Survey participants were queried on the percentage of the total assessment that could be derived from online assessments for each of the units they teach. Forty-two per cent of respondents reported on the percentage of online assessment they use in a total of 264 units (details of the percentage of the unit assessable online were only provided for 21 per cent of the units represented by the responses). A graph of the distribution of the percentage of each unit assessable online, in deciles (groups of 10 per cent), is presented in Figure 1. 0 10 20 30 40 50 60 70 80 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% N u m b e r o f u n i t s Figure 1: Percentage of each unit assessable online (n = 264) Figure 1 shows that the largest deciles consist of units that utilise some form of online assessment for only up to 20 per cent of the total unit mark. It can be seen that the majority of units (81%) only utilise a small to moderate quantity of online assessment (less than 50 per cent). The survey also asked participants to reveal the number of units they teach that utilise Web supplemented online coursework, the number of units that utilise a partial amount of online assessment, and the number of units that are offered fully online. These results are shown in Table 4. These results show that 71 per cent of the respondents conducted units with some form of online component, indicating that overall the use of technology is quite widespread. However, the use of online assessment is not as widespread, being used in far less units than online supplementary coursework, with Byrnes and Ellis 113 units that utilise online assessment for 100 per cent of the unit assessment even fewer. This is in line with findings by Hill (2002, quoted in Booth, et al, 2003: 18) who states, “there are few examples of fully online courses”. Table 4: Respondents’ use of online assessment Level of assessment Percentage of respondents using Number of units using Online coursework only 71% 780 Partially online assessable 43% 357 Fully online assessable 16% 139 Question 2: Does the use of online assessment vary with university size and age? Universities were compared by grouping them based on whether they are big universities (more than 17,000 enrolled students) or small universities (less than 17,000 enrolled students), and whether they are old universities (formed before 1975) or new universities (formed after 1975). These figures were chosen so as to separate the universities in the sample into approximately equally sized groups. Statistical tests showed no significant differences between old and new universities with regard to either the use of partially online assessable units or the use of fully online assessable units. Neither were significant differences found between large and small universities with regard to either the use of partially online assessable units or the use of fully online units. It would therefore appear that the use of online assessment does not vary with university size or age. Question 3: What types of online assessment are in use? Survey participants were asked whether they used online quizzes, whether they used online forums and whether they used an online drop box for assessment purposes. The use of an online assignment drop box was not considered a pure form of online assessment, as it is merely an alternative means of delivering a traditionally prepared assessment. However, it is considered an important stepping-stone to fully online assessment and was therefore included in this survey. There are still a large number of respondents (44%) who use none of these online assessment methods. The results are summarised in Table 5. Statistical tests indicated that online assignment drop boxes are significantly more common than online forums, and that online forums are significantly more common than online quizzes. 114 Australasian Journal of Educational Technology, 2006, 22(1) Table 5: Types of online assessment in use (individual respondents may use more than one method) Assessment method Percentage of respondents using Online quizzes 22% (n = 388) Online forums 30% (n = 384) Assignment drop box 39% (n = 388) None of the above 44% (n = 381) Participants were also asked to comment on other forms of online assessment that they utilise. These comments were freeform text fields, which are condensed and categorised in Table 6. The reason that these were not specifically queried was that a review of the literature indicated that the majority of online assessment consisted of online quizzes, online forms and online assignment drop boxes, and that other forms of online assessment were both varied in nature and poorly utilised. It was considered more appropriate to query what additional types of online assessment were in use in a freeform manner that allowed respondents the flexibility to specify their exact methods rather than being constrained to pre-conceived ideas of what they might be using. Table 6: Forms of online assessment other than online quizzes, online forums, and online assignment drop boxes (n = 391) Assessment method % of respondents using Peer review/assessment 1.8% Email submission of assignments 1.3% Chat sessions 1.3% Online portfolios 1% Use of simulation software 0.8% Online group collaboration 0.8% Creation of Web page 0.5% Online presentations 0.5% Online computer programming tests 0.5% Online Web searching 0.3% Online lab report 0.3% Online journals 0.3% Online role play 0.3% Creation of digital video clip 0.3% It should be noted that very few respondents reported using any alternative methods of online assessment. In Table 6, the value of 0.3 per cent relates to just one response. Several respondents reporting used more than one alternative form of online assessment, however the majority of the respondents represented in Table 6 reported utilising only one alternative. Email submission of assignments is another, less integrated form of the assignment drop box. Chat sessions are closely related to forums, with the Byrnes and Ellis 115 difference being that chat sessions allow synchronous communication while forums are asynchronous. The findings of this study clearly indicate that beyond the use of online quizzes and online forums, there is very little use of other online assessment strategies such as the creation of online portfolios, simulations, role playing and games. Question 4: How prevalent are online summative final exams? Information was also gathered as to the use of online summative final exams. Information on the use of final exams with partially online assessable units was provided for 300 of the 357 partially online assessable units. The other 57 partially online units not accounted for were due to a number of incomplete and a few incorrectly completed responses. Only six per cent of partially online assessable units had a final exam that could be taken online. Of the partially online assessable units without an online final exam, 83 per cent had a traditional pen and paper final exam, indicating that the use of final exams is quite widespread. Overall, 84 per cent of partially online assessable units had a final exam. Information on the use of final exams with fully online units was provided for 95 of the 139 fully online units. The other 44 fully online units not accounted for were due to a number of incomplete and a few incorrectly completed responses. Of the 95 fully online units accounted for, only seven per cent had a final exam. These results are summarised in Table 7. Table 7: Percentage of units with online, traditional and no final exams Percentage online exam Percentage traditional exam Percentage no exam Partially online assessable units (n=300) 6% 78% 16% Fully online assessable units (n=95) 7% 0% 93% It is clear that the use of final online exams with fully online units is not widespread. The data revealed that some partially online assessable units do have a final exam online, but these represent only six per cent of the exams available in these types of units. Overall, the use of online summative final exams are quite rare, whereas the use of more traditional pen and paper final exams is quite widespread. Question 5: Is the use of online assessment related to discipline area? Two comparisons were performed between major discipline areas and the presence of (1) the number of partially online assessable units, and (2) the 116 Australasian Journal of Educational Technology, 2006, 22(1) number of fully online units, to determine whether there is any significant difference between the use of online assessment across disciplines. Statistical tests revealed that Arts; Engineering; Biological Science; Economics and Accounting and Health Sciences and Psychology had very low levels of fully online assessable units. Business, Mathematics and Information Technology and Law had slightly higher levels of fully online assessable units. Education and Communication; Hospitality and Tourism; Physical, Chemical and Environmental Science; Marketing and Management; Nursing and Social Science and Humanities had higher levels of fully online assessable units. No significant difference, however, was found between discipline area and the presence of partially online assessable units. Question 6: What is the perceived difficulty of supporting mixed modes of delivery? Survey participants were asked to rate how difficult they considered mixed mode delivery was to support. The responses are summarised in Figure 2. 0 10 20 30 40 50 60 70 80 much easier bit easier the same bit harder much harder N u m b e r o f r e s p o n s e s Figure 2: Rating of difficulty of mixed mode delivery (n = 240) Thirty three per cent of respondents (n=240) considered mixed mode delivery to be either much easier or a bit easier than single mode of delivery. Fifteen per cent considered the difficulty level to be the same. Fifty two per cent considered mixed mode delivery to be either a bit harder or much harder than single mode of delivery. Byrnes and Ellis 117 Although this result is statistically significant, further tests showed that there was no relationship between respondents’ ratings of this item and their use of partially or fully online assessable units, leading to the conclusion that a respondent’s feeling of how difficult supporting mixed mode delivery is does not significantly predict their use of online assessment. Question 7: Should all units be provided fully online? Survey participants were questioned as to whether they felt that all units should be provided fully online. Three hundred and seventy-four respondents answered this question. Of these, 66 per cent believed that not all units should be available fully online, 25 per cent believed that all units should be fully online, and 9 per cent did not know. Statistical tests indicated that respondents with some units fully online are more likely to believe that all units should be online, but no causal relationship could be determined. That is to say, respondents may have come to the conclusion that all units should be fully online after creating one or more fully online units; or they may have created fully online units based on their existing belief that all units should be fully online. Question 8: What is the impact of available learning management systems on online assessment? Survey participants were asked whether there was a centralised LMS available for their use. Of the 388 responses received for this question, 14 per cent said there was not, 70 per cent said there was and 16 per cent did not know. For the 34 institutions represented in this survey, results from a separate survey of LMS installations at Australian higher degree granting institutions (Byrnes & Ellis, 2004) were obtained. Table 8 shows the distribution of LMSs in use at the institutions represented in the responses to this survey (n=34). Table 8: Distribution of learning management systems in use at Australian institutions LMS Percentage of institutions No LMS 6% BlackBoard 35% WebCT 44% In-house 15% 118 Australasian Journal of Educational Technology, 2006, 22(1) When these results were compared with the 14 per cent of respondents (n=388) who indicated that there was no LMS on their campus, it became clear that only five were correct. Seventeen had access to BlackBoard, 21 had access to WebCT, and nine had access to an in-house product. These were distributed fairly evenly across the sample of universities, with 26 universities (out of 34) containing respondents who were unaware of their central LMS. Two hundred and eighty-three responses indicated how the respondent found using the central LMS. Twenty point five per cent indicated that they had not used it, 10 per cent indicated they did not like it, while 44.5 per cent responded that it was okay and 25 per cent responded that it was good. Statistical tests showed no significant difference between these ratings based on the LMS product in use, indicating that respondents in general did not prefer one particular LMS product over another. This result should not be seen as indicating that all LMS products are equivalent, as the products were not compared against each other. Additionally, no significant difference was found when comparing the LMS product in use with the use of partially online assessable units. A significant difference was found when comparing the LMS product with the use of fully online assessable units – fully online assessable units were more prevalent in a BlackBoard environment than in an environment with an in-house LMS. The lack of further significant differences, however, would indicate that the use of different LMSs does not accurately predict the use of fully online assessable units. Question 9: What is the impact of available support groups on online assessment? Seventy-eight per cent of respondents reported that there was a support group who could help them with putting materials online. Six per cent responded that there was no support group, and 16 per cent did not know. Although different universities can have widely different names and structures for their support groups, the groups identified in this survey can be categorised into the following six types: • Support provided by the Information Technology Unit. • Support provided by a Teaching and Learning Unit. • Support provided by a Flexible Delivery Unit. • Support provided by the school or faculty. • Product support group specifically set up for the particular LMS product in use. Byrnes and Ellis 119 • Other types of support not covered by the above five categories. This group may consist of an individual who may be consulted, and is completely informal in nature. Each survey participant was asked to rate each of the support groups available to them. A rating of zero was specified if the respondent had not used the support group, otherwise the support group was rated from one (no use at all), to five (indispensable). The summary of ratings across the six categories of support group is shown in Table 9. Table 9: Summary of ratings for support groups (scale of one to five) Support Group n Mean S.D. Information Technology 32 4.02 1.111 Flexible Delivery 57 4.13 1.046 Teaching and Learning 86 3.42 1.046 School 15 4.20 0.941 Product 43 3.86 1.146 Other (e.g. individual) 28 3.62 1.160 Statistical tests revealed that the only significant difference existed between the scores for the Flexible Delivery support group and the Teaching and Learning support group. As the mean score for the Flexible Learning support group is higher, it is concluded that respondents are more satisfied with the support provided by this group than with the Teaching and Learning support group. Given that the precise nature of the groups in the survey that were categorised under Flexible Delivery and Teaching and Learning is unclear, it is difficult to draw any definite conclusion from this result. It is possible that other significant differences would have been found between other support groups had the sample size been larger. Further analyses were performed to determine whether there were any significant differences between the level of use of partially online assessable units or fully online assessable units based on the ratings given to the support group. No significant differences were found, leading to the conclusion that the respondents’ rating of the support group has no influence on their use or lack of use of online assessment. Question 10: What other institutional factors are involved in the use of online assessment? As a final component of the survey, respondents were invited to include any additional comments they felt were appropriate to the research. Discussions of the main themes present in the comments that are most pertinent to this question are included below. 120 Australasian Journal of Educational Technology, 2006, 22(1) Twenty-one respondents commented that the time and effort required to put units online can be prohibitive, which represents the largest number of comments on any one issue. Obviously, the move to greater levels of use of online assessment is becoming a heavy burden on academics, taking time away from their core responsibilities of teaching and research. Two respondents specifically mentioned that multi-mode teaching is especially difficult, and three respondents noted that if a unit is updated regularly, the use of online coursework could be prohibitive. In conjunction with these workload related issues, 20 respondents mentioned the issue of support – five commenting that good support that helps assemble material is essential, and 15 mentioning that currently available levels of support are difficult to access, of poor quality, or costly (to the school). Seven respondents made mention of the fact that the university’s central administration pushed the use of teaching online, but failed to back that up with adequate funding. One respondent noted a definite lack of commitment at the university level for online assessment. Five respondents commented on technology issues related to central services. Two respondents mentioned that software failure of LMSs is problematic, one noted that the “LMS system changes often and is counter- intuitive”, one commented on “security concerns with central services” and one mentioned that generally “technology problems can be an issue”. These comments indicate concerns with the reliability of central LMS services. Eight respondents cited pedagogical issues with relation to the provided technology. Four commented on the lack of flexibility of central services and LMSs and two mentioned that the potential to use online assessment is there if software is improved. These comments show that the greatest impacts on academics’ use of online assessment are workload related. This is confirmed by the number of comments related to adequate support, but also showed that many academics do not consider the available support group(s) provided by their institution to be adequate. Additionally, there is a definite view that the impetus to use online assessment comes from central administration, but that this is not backed up with either adequate funding or commitment, either in terms of considering academics’ workload related issues or in terms of providing adequate technology. Byrnes and Ellis 121 Conclusion This paper has explored and mapped the current use of online assessment in Australian universities. The results showed that the use of online assessment, although quite widespread, is largely superficial. Less than 40 per cent of the units reported on in the survey responses used any form of online assessment. Of these, the majority utilised online assessment for only 10-20 per cent of the total unit grade. Less than 11 per cent of units were fully assessable online. It should further be noted that these results are likely to be biased in favour of online assessment. Although the survey attempted to be as representative as possible, it is likely that there was a greater level of response from those individuals already involved in, or with an interest in, online assessment. Therefore these results should be viewed as representing the absolute maximum amount of online assessment in use, with the likelihood that in reality it is less. The results failed to show any difference in the use of online assessment between larger and smaller universities, or between older, better established universities, and younger, less well-established universities. It has been mentioned that the less well-established universities have the greatest need of establishing online programs (Dean, 2002). These results indicate that they are not competing effectively in this area, whereas they should be carving out a niche in this new and evolving sector. In terms of the types of online assessment in use, the results showed that beyond the use of online quizzes and online forums, there is very little reported utilisation of other types of online assessment. There is research to indicate that other forms of online assessment such as portfolios, role plays and simulations have great potential (Reeves, Herringon & Oliver, 2002; Oliver, 2001; Kendle & Northcote, 2000), especially in conjunction with online quizzes (Honey & Marshall, 2003). More exposure to these forms of non-traditional assessments may be needed before they are widely adopted. This research showed that partially online units tend to utilise a traditional pen and paper final exam at the end of the teaching period. This tradition is not reflected in the current evolution of fully online units. There are still a number of issues with regard to the use of online exams such as invigilation, authentication and equity of access that are most likely responsible for the small number of units with a final exam online. However, there is also research to suggest that a final exam is not necessarily an appropriate component of good assessment practice (Phillips & Lowe, 2003: 419; Race, 2003), and the move away from this type 122 Australasian Journal of Educational Technology, 2006, 22(1) of approach could be seen as a positive outcome. Further research into this area would be required in order to shed more light on these results. There was an indication that different disciplines used different amounts of online assessment. Interestingly, the disciplines that use less online assessment would appear to be somewhat more suitable to the use of online assessment, and it is perhaps for other reasons, such as a greater focus on education within the discipline, that this difference is present. Whatever the case, using discipline area as an excuse to not use online assessment does not appear to be valid. Although more respondents indicated that they found supporting mixed modes of delivery to be more difficult than a single mode of delivery, this did not predict their use of online assessment. It is possible, however, that the quality of online assessment is affected, but further research would be required to determine this. The majority of respondents indicated that they didn’t consider that all units should be available fully online. More research would be needed to determine why this is so, however there was some indication that academics who already had units fully online were more likely to think that all units should be provided fully online. It may be a matter of resistance to new technology that has resulted in such a disparity. There is research to show that fully online units can be more effective than their traditional counterparts (Morris & Zuluaga, 2003). Interestingly, the LMS in use did to some extent predict the use of online assessment, with users of BlackBoard more likely to employ fully online units than in-house LMSs. Anecdotally, BlackBoard is easier to use, and this may explain this result. However, the results were not conclusive, and it would be incorrect to state from this research that any one LMS product is better than another. Although there was a significant difference between ratings given to different types of support groups, this did not predict the use of online assessment. It is concluded that the quality of the support group does not affect the use of online assessment, although it may affect the quality of the online assessment. Although not supported by the statistics due to the low numbers involved, it would appear that support groups that have a focus on online delivery (flexible learning support groups) and support groups that are accessible (school based support groups) are the most well regarded and that institutions should use these findings when planning and implementing their own support structures. The additional comments provided by the respondents shed more light on some of these results. A number of respondents commented that in their Byrnes and Ellis 123 institution the impetus to use online assessment comes from central administration, not from a pedagogical need or advantage. However, there is also a lack of funding and resourcing associated with this push, and academics are required to do much of the work in their own time with limited support. This has given rise to a large number of workload related issues, with academics barely finding the time to teach, research and implement their units online, let alone undertake the pedagogical restructuring required to implement quality online coursework and especially assessments. This helps to explain the widespread but largely superficial use of online assessment that was observed, as an attempt by academics to satisfy institutional demand yet at the same time mitigate the effects of increased workload. If institutions are to take online delivery of coursework seriously, they also need to have a heightened awareness of the importance of online assessment as a fundamental component of that process. Accessible support groups that are familiar with the pedagogy and technology of online assessment, that can facilitate the development of online assessments and reduce the workload of academic staff, are essential. References Australian Bureau of Statistics (2004). [viewed 12 Jan 2006] http://www.abs.gov.au/Ausstats/abs@.nsf/0/E6C05B29C52B332DCA256DEA 00053A4D?Open Australian Flexible Learning Framework Quick Guides (2002). What have we learnt about … Online Assessment. [viewed 12 Jan 2006] http://flexiblelearning.net.au/guides/assessment.html Black, P. J. (1998). Evaluation and assessment. In Connecting research in physics education with teacher education. International Commission on Physics Education. [viewed 12 Jan 2006] http://www.physics.ohio- state.edu/~jossem/ICPE/E2.html Booth, R., B. Clayton, B., Hartcher, R., Hungar, S., Hyde, P. & Wilson, P. (2003). The development of quality online assessment in vocational education and training. Volume 1, National Centre for Vocational Education Research. [viewed 12 Jan 2006] http://www.ncver.edu.au/publications/962.html Buchanan, T. (2000). The efficacy of a World-Wide Web mediated formative assessment. Journal of Computer Assisted Learning, 16, 193-200. Bull, J. & McKenna, C. (2004). Blueprint for computer-assisted assessment. London: RoutledgeFalmer. Byrnes, R. (1994). The development and evaluation of a computer-aided assignment management system, Unpublished Honours Thesis, Southern Cross University, Australia. 124 Australasian Journal of Educational Technology, 2006, 22(1) Byrnes, R. & Ellis, A. (2004). The distribution and features of learning management systems in Australian universities and their role in student assessment. In Proceedings of AusWeb04, Gold Coast, Australia. [viewed 12 Jan 2006] http://ausweb.scu.edu.au/aw04/papers/refereed/byrnes Cochran, W. G. (1963). Sampling techniques. 2nd Ed., New York: Wiley. Curoe, R. (1999). Instructional design in a technological world: Fitting learning activities into the larger picture. Journal of Interactive Instructional Development, Fall: 23-27. Dean, T. (2002). Telelearning: Invention, innovation, implications. Towards a manifesto. Electronic Journal of Instructional Science and Technology, 5(2). http://www.usq.edu.au/electpub/e-jist/docs/Vol5_No2/Dean%20- %20Final.pdf [viewed 12 Jan 2006] Department of Education, Science and Training (2002). Universities online: A survey of online education and services in Australia. Commonwealth of Australia. [viewed 3 Jan 2005] http://www.detya.gov.au/highered/occpaper/02a/default.htm Erdinc, Z. (2002). Australia online: Borderless university. Turkish Online Journal of Distance Education, 3(4). [viewed 12 Jan 2006] http://tojde.anadolu.edu.tr/tojde8/articles/australiaonline.htm Gallagher, M. (2001). E-Learning in Australia: Universities and the new distance education. In Proceedings of the 7th OECD/Japan Seminar: E-Learning in Post- Secondary Education. Tokyo. [viewed 12 Jan 2006] http://www.oecd.org/dataoecd/16/52/1854142.pdf Harris, R. A., Beevers, C., Ewins, R. Haywood, J. & McAteer, E. (2002). Encouraging research into on-line learning and assessment. In Winds of change in the sea of learning. Proceedings ASCILITE 2002, Auckland, New Zealand. [viewed 12 Jan 2006] http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/158.pdf Honey, M. & Marshall, D. (2003). The Impact of on-line multi-choice questions on undergraduate student nurses’ learning. In G. Crisp, D. Thiele, I. Scholten, S. Barker & J. Baron (Eds), Interact, Integrate, Impact: Proceedings ASCILITE 2003. Adelaide, 7-10 December (pp. 236-243.) [viewed 12 Jan 2006] http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/236.pdf James, R., McInnis, C. & Devlin, M. (2002). Assessing learning in Australian universities. Centre for the Study of Higher Education, University of Melbourne, Australia. [viewed 12 Jan 2006] http://www.cshe.unimelb.edu.au/assessinglearning/ Kerka, S. & Wonacott, M. E. (2000). Assessing learners online. ERIC Clearinghouse on Adult, Career and Vocational Education Practitioner File. [viewed 12 Jan 2006] http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServle t?accno=ED448285 Kendle, A. & Northcote, M. (2000). The struggle for balance in the use of quantitative and qualitative online assessment tasks. In Proceedings ASCILITE 2000, Coffs Harbour. [viewed 12 Jan 2006] http://www.ascilite.org.au/conferences/coffs00/papers/amanda_kendle.pdf Byrnes and Ellis 125 McKey, P. (1997). The development of the online educational institute. Unpublished Masters of Training and Development Thesis, School of Social and Workplace Development, Southern Cross University, Lismore. Morris, E. J. S. & Zuluaga, C. P. (2003). Educational effectiveness of 100% online I.T. courses. In G. Crisp, D. Thiele, I. Scholten, S. Barker, & J. Baron (Eds), Interact, Integrate, Impact: Proceedings ASCILITE 2003. Adelaide, 7-10 December. (pp.353- 363.). http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/353.pdf [viewed 12 Jan 2006] Oliver, R. (2001). Strategies for assuring the quality of online learning in Australian higher education. In Proceedings of Moving Online II Conference, Surfers Paradise, Queensland. http://elrond.scam.ecu.edu.au/oliver/2001/mocpaper.pdf [viewed 12 Jan 2006] Peat, M., Franklin, S., Devlin, M. & Charles, M. (2004). Revisiting associations between student performance outcomes and formative assessment opportunities: Is there any impact on student learning? In Beyond the comfort zone: Proceedings ASCILITE 2004. Perth, Western Australia, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/peat.html Phillips, R. & Lowe, K. (2003). Issues associated with the equivalence of traditional and online assessment. In G. Crisp, D. Thiele, I. Scholten, S. Barker, and J. Baron (Eds), Interact, Integrate, Impact: ProceedingsASCILITE 2003. Adelaide, 7-10 December 2003: (pp.419-431). [viewed 12 Jan 2006] http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/419.pdf Race, P. (2003). Why fix assessment? – a discussion paper. [viewed 12 Jan 2006] http://www.scu.edu.au/services/tl/why_fix_assess.pdf Reeves, T. C., Herrington, J. & Oliver, R. (2002). Authentic activities and online learning. In Proceedings HERDSA 2002: 562-567. [viewed 12 Jan 2006] http://elrond.scam.ecu.edu.au/oliver/2002/Reeves.pdf Senate Inquiry in to Higher Education (2001). Universities in crisis. [viewed 12 Jan 2006] http://www.aph.gov.au/senate/committee/eet_ctte/ completed_inquiries/1999-02/public_uni/report/contents.htm White, S. & Davis, H. (2000). Creating large-scale test banks: A briefing for participative discussion of issues and agendas. In Proceedings of the Fourth International Computer Assisted Assessment Conference, Loughborough, United Kingdom. http://s-d.lboro.ac.uk/caaconference/conf2000/pdfs/whites.pdf [viewed 3 Jan 2005] Rod Byrnes, School of Social Sciences Southern Cross University, PO Box 157, Lismore NSW 2480 Australia Email: rod.byrnes@scu.edu.au Allan Ellis, School of Commerce and Management Southern Cross University, PO Box 157, Lismore NSW 2480 Australia Email: allan.ellis@scu.edu.au