heinrich.pdf Australasian Journal of Educational Technology 2012, 28(2), 279-294 Pathways for improving support for the electronic management and marking of assignments Eva Heinrich, John Milne Massey University Bruce Granshaw Victoria University of Wellington This article investigates the support e-learning can provide for the management and marking of assignments. The work is contextualised in the importance of assessment with assignments in tertiary education, in the theories about high quality marking of assignments, and the practical experiences of academics at tertiary institutions. The tasks that need to be carried out as part of assignment management and marking are detailed. A set of requirements is derived based on the education literature and on practical needs established in interviews with academics. Possible architectures for an e-learning system implementing these requirements are suggested and arguments are provided towards using an architecture that combines existing learning management system functionality for assignments with a specialised assignment management and marking application. A specific implementation based on Moodle and Lightwork is introduced. The article concludes with thoughts on the cognitive fit of learning management systems for assignment marking tasks and their core roles in supporting teaching and learning. Introduction This article describes work undertaken to address the research objective of improving support for the management and marking of electronically submitted assignments. The goals of the research are to build on education theories, to utilise e-learning approaches, to improve efficiency around assignment marking, and to improve the quality of feedback given to students. The research reported here builds on earlier work undertaken by the research team that included an extensive literature review (Heinrich, Milne, Crooks, Granshaw & Moore, 2007), interviews with 90 academics across several tertiary institutions ( Milne, Heinrich, Ramsay, Morrison, Granshaw & Moore, 2007), and a review of e-learning tools for support of assignment marking (Moore, Granshaw, Milne & Heinrich, 2007). Aspects of this work have also been reported in Heinrich, Milne, Moore (2009) and Heinrich, Milne, Ramsay and Morrison (2009). The article starts by reviewing briefly the foundations of assignment marking given in the education literature and by looking at what is reported in the literature on e- learning support for assessment. From there the article moves to propose a set of requirements for electronic marking that have been derived from the literature and from interviews with academics. Potential architectures for e-learning solutions are discussed. These are based on learning management systems (LMS), generic software 280 Australasian Journal of Educational Technology, 2012, 28(2) and specialist marking systems. After making an argument for LMS-based solutions a further distinction is made between pure LMS solutions and solutions that combine LMS with a specialised application. Strengths and weaknesses of both options are discussed and the experiences with a combined solution, featuring the LMS Moodle and the specialised application for management and marking of assignments called Lightwork, are outlined briefly. Foundations Assessment plays a central role in education and is integral to the achievement of educational goals, course design and student motivation (Heinrich et al., 2007). Assessment is tightly linked to the teaching and learning strategy of a course (Kendle & Northcote, 2000; Macdonald, 2003; Muirhead, 2002). Tertiary institutions are increasingly under pressure to enhance the quality of their assessment (Hattie, 2009). The education literature provides strong theoretical foundations for assessment. Principles to guide the marking of assignments are well established and are based on marking guidelines, rubrics and moderation (Brown, 2009; Gronlund, 2006; Hanna & Dettmer, 2004; Lambert & Lines, 2000; Linn & Miller, 2005; Nitko, 2004). Analytic approaches to marking derive grades from looking at the criteria specified in a marking rubric, whereas holistic approaches make a quality judgment, expressed via a grade first and then look for justification based on criteria (Sadler, 2009). While the analytic approach is commonly applied in higher education today, arguments for reconsideration of assessment based on a holistic approach have been brought forward (Sadler, 2009). Appropriate feedback is essential for supporting student learning (Hattie, 2009). Specific feedback to students needs to make explicit what good work looks like and how well it matches the desired goals (Nitko, 2004; Sadler, 1989). Students appreciate highly feedback on their work (Margrain et al., 2009). E-learning solutions for assessment are becoming increasingly important. Whitelock and Watt (2007) concluded that technology can be used effectively to enhance assessment of student submissions, providing there is awareness that primarily it should support and improve student learning. They describe technology based assessment tools with the capacity for delivering feedback electronically. Previous work by Whitelock and Brasher (2006) identified enablers for effective adoptions of e- learning solutions as institutional support and staff professional development. They acknowledged a “call for a pedagogically driven model for e-Assessment as part of a vision for teaching and learning in 2014” (p.7). Dermo (2009) investigated student perceptions of the use of e-learning tools in a British university. The factors examined were affective factors, validity, practical issues, reliability, security and learning and teaching. Findings indicated generally positive attitudes towards these factors with students stating that electronic assessment contributed positively to the teaching and learning process. Electronic support for the submission, management and marking of assignments provides administrative benefits, time and resource savings, and the removal of geographical limitations (Milne et al., 2007). Advantages include the support for learning with respect to the timeliness and quality of marker feedback; increased accessibility; consistency of quality of feedback due to the ability of markers to re-mark Heinrich, Milne and Granshaw 281 or parallel mark assignments; and possible participation of students in an electronic peer assessment process (Whitelock, 2009). E-learning assignment systems aim to provide better feedback to students, increase efficiency and effectiveness and support analysis of learning outcomes (Burrows & Shortis, 2011). Denton, Madden, Roberts and Rowe (2008) investigated the effectiveness of electronic formative feedback by using control groups and found that students rated feedback given electronically as being superior. There was more clarity with the marking scheme; legibility of feedback was greater; and there was better information on deficient and good aspects of the submission. Further to this, markers were able to return work more quickly and with high confidence in their levels of fairness and balance. Figure 1 summarises key points from the literature regarding the importance of assessment for learning, characteristics of assignment marking and electronic support. Importance of assessment for learning Assessment has a strong influence on student learning. Written student responses such as essays or reports, are important for deep learning. High quality formative feedback strongly contributes to raising student achievement. Characteristics of assignment marking Assignment marking should be based on clearly specified marking rubrics. Feedback needs to be individualised, include comments on higher level aspects, explain characteristics of high quality work and show progression pathways. Carefully constructed comment banks aid the marking process. Assignment marking commonly includes summative and formative components. Assignments need to be marked by human markers instead of automated processes. Assignment marking is time consuming. Achieving of validity and reliability is challenging. Inter- and intra-marker consistency needs to be monitored. Marked work needs to be returned to students within a short timeframe. Electronic support for assignment marking Electronic assignment submission and return of marking has advantages in terms of speed, independence of physical location and document management. Marking can be completed faster as less time has to be spent on administrative tasks. The consistency and quality of feedback can be better, due to increased opportunities for marker guidance, comparisons and reviews of feedback. The feedback provided to students is likely to be more complete, better structured and more legible. Figure 1: Key points from the literature 282 Australasian Journal of Educational Technology, 2012, 28(2) Requirements for electronic solutions As outlined in the previous section the education literature refers to marking rubrics, high quality feedback and consistency checking as key ingredients of ‘good’ marking. An e-learning solution therefore should provide strong support for these factors. Assignment marking also needs to address a range of practical issues, such as viewing the student work, the safekeeping of this work, the distribution to markers and the return of marking results to students. A schedule of tasks for dealing with assignments has been developed based on the practical issues derived from interviews with academics (Milne et al., 2007), and a number of education theories. This schedule is presented in Figure 2. Setting up the assignment Creating assignment drop box; specifying parameters like due date and submission options; making task specification available Handling submissions of student work Storing files students have submitted; recording submission parameters like date and time Pre- marking Marking preparation Creating marking rubric including comment banks; allocation of markers to students Marking Reading student work Producing detailed feedback to students aligned to criteria specified in the marking rubric Deciding on marks or grades aligned to criteria specified in the marking rubric Monitoring progress Checking on progress of marking to facilitate timely return to students Quality control Reviewing the quality of individual marking; looking at inter- and intra-marker consistency Marking and quality assurance Release to students Making the marking result comprised of marks and feedback available to students Bookkeeping of marks Retaining assignment marks as these will contribute to overall course grades Reflection on assignment Reflecting on task specification, and marking in light of desired learning outcomes Post- marking Figure 2: Tasks related to assignment management and marking Heinrich, Milne and Granshaw 283 The tasks start with what can be called the ‘pre-marking’ tasks. These include setting up the assignment and receiving, storing and bookmarking the submissions of student work. The pre-marking tasks are followed by core tasks for marking and quality assurance. The marking rubric needs to be set up and markers need to be allocated to students. Feedback and marks need to be assigned to student work, based on the criteria specified in the marking rubric. The progress of marking needs to be monitored to achieve timely feedback and the quality and consistency of marking has to be checked. These core tasks are followed by what can be called ‘post-marking’ tasks. These focus on releasing marking results to students and keeping track of marks for grade calculations. Reflections by the lecturer on the assignment and its marking conclude the process. The key requirements for an e-learning system can be specified based on the tasks related to assignment management and marking. These can be divided into requirements that build on educational theories and into non-functional requirements that arrive from tertiary practice. Building on educational theories Following an analytic approach to marking, marking rubrics are at the centre of educationally sound marking. Marking rubrics define criteria that the students need to address in their work and which form the basis for the assessment. Marking rubrics can specify quality levels in relationship to the criteria. An effective assignment assessment e-learning tool needs to provide a range of rubric-related functionalities. Teachers need to be able to define a marking rubric for each assignment. Such a rubric will contain criteria with descriptions directed at students. They might also contain descriptions directed at markers for guidance and comment banks associated with the criteria. Depending on the type of marking rubric, there will also be quality levels for the criteria, with marks and weightings associated with criteria and quality levels. It needs to be possible to exchange the marking rubric between the members of a marking team to facilitate a pre-marking quality review and the actual marking. In preparation for marking, copies of the marking rubric need be created for each student. These individual copies will later contain the student specific feedback and marks. The markers need to be able to draw on the comment banks, either using pre- defined comments directly or by individualising comments. Marker allocation forms the basis for progress checking and quality review. The following functionality is required. All potential markers for an assignment have to be identified, based on their access rights to the course. The teacher has to be able to allocate markers to students. Depending on course context this will happen using either predefined allocation algorithms or manual allocation of groups or individuals. The marking allocation has to be shared across the members of the marking team. As marking advances an overview on marking progress should be shown. This will include highlighting the status of marking, and different perspectives with focus on students, marker allocations or submission times. The teacher must be able to access the marking completed by the markers to enable quality review. Marking that needs further work has to be returned to markers or re-allocated to a new marker. Marking that passes the quality review has to be labelled accordingly, to be released to students 284 Australasian Journal of Educational Technology, 2012, 28(2) at a point of time determined by the teacher, for example, when all marking for the class has been completed. It needs to be possible to review the quality of feedback and the suitability of marks awarded prior to the return of marking to students. Marking results should be converted into a generic format that can be read by students without the need for specialised software. Students should be able to extract the marking feedback, for example in separate feedback sheets, to facilitate access beyond course duration. To assist the teacher with reflection after the assignment marking is completed, there should be functionality for a detailed review. This will include extracting marks and comments for statistical and textual analysis. These data will be separated by criteria and markers to allow for comprehensive analysis. For example, knowing statistics on the marks per criterion can help the teacher to identify student strengths or weaknesses in specific areas of the assignment task. Teachers need to be able to retain access to marking rubrics beyond course duration, as these rubrics can serve as templates for use in future assignments. After reflection on the completed assignments, comment banks can be revised and improved for future assignments on conceptually similar tasks. Non-functional requirements The following non-functional requirements should be satisfied by the e-learning system. Across the tertiary sector there is a huge variety of assignment contexts, stemming from differences in disciplines, class sizes or levels of study. The system should cater for a wide range of contexts. The system should allow for semi-automated processes, as for example in the marker allocation process for large classes, but also be able to facilitate individual attention, as might be suitable for small student numbers. It is acknowledged that some discipline specific requirements, such as the annotation of student work with mathematical symbols, might not be covered by a generic tool and might require specific add-ons. Assessment data are sensitive and need to be protected from unauthorised access. The tool must ensure that only authorised markers can access student work and marking, and that all data are kept secure. The communication and exchange between members of the marking team has to be secure. Marking data need to be safely stored to avoid data loss. Academics have expressed that they like to mark at different locations. For example, an academic might start marking at the office computer and later continue working from home. The system needs to support this, providing seamless transitions from one work location to the next. Related to this is the requirement to be able to work offline, away from the continuous presence of a network connection. The access to data needs to be efficient, especially in the contexts of large classes and potentially slow network connections. The system needs to integrate with other solutions that provide support for assignment assessment. For example, checking for plagiarism is an area for which specialised solutions exist and are widely used. Plagiarism checking functionality should not have to be re-implemented. Connections to systems for the institution-wide Heinrich, Milne and Granshaw 285 processing and storage of marks and grades need to be seamless. The architecture of the system should allow for future extensions in assignment marking functionality. The tool should increase the efficiency of marking. It should automate steps that can be performed by a computer, such as repeated copy and paste to create feedback sheet templates for each student. The system should impact on the quality of marking by making concepts from the education literature on assignment assessment easily accessible. The system should follow accessibility guidelines and its outputs, such as feedback sheets given to students, need to be fully accessible. The system should be easy to learn and operate, allowing teachers and markers to focus on the marking tasks without being distracted by system characteristics. Solutions based on LMS, generic software and specialist systems The task descriptions and requirement specifications show that a system supporting the management and marking of assignments needs tools for document exchange and storage, for capturing marker allocations and progress checking, for creating marking rubrics and creating feedback documents for students. These requirements can be achieved, with varying degrees of success, by using an LMS, generic software and specialist systems. Each approach has advantages and limitations. It is possible for teachers to create a ‘system’ based on standard generic software. Documents can be exchanged via email and can be stored on local or shared file systems. Class lists can be kept in spreadsheet documents and the marker allocations can be entered there. Markers can send updates on their progress via email to the teacher in charge. Word processing software can be used to create marking rubrics. Feedback sheets for each student are created via copying and pasting the rubric document and student details. It is easy to see that this approach has a number of disadvantages. There are great inefficiencies in having to perform the tasks described in this semi-manual form. The larger the class size the more time will be wasted. There is a high risk for errors occurring. For example, it could easily happen that results are recorded against or sent to the wrong student. There is no guidance on good marking practices. Teachers have to create their own ways of dealing with marking and quality control. The interviews with academics referred to in the introduction to this article have provided many examples of the problematic around these issues. More sophisticated approaches to create solutions outside LMS have been attempted. Denton et al. (2008) have constructed a feedback system that is based on Microsoft Office and extends this standard software by supporting the construction of marking feedback. Denton et al. write about the problems of building on generic software that changes frequently, affecting the add-on functionality. GradeMark is a fully online assignment marking and management system [http://turnitin.com/static/prod ucts/grademark.php]. It provides functionality to make annotations and is based around a rubric. It has seamless integration with the plagiarism checker Turnitin and grades can be exported to a spreadsheet for input into the institutional result processing systems. While plagiarism checking with Turnitin can be triggered from submission of student work to an LMS, The use of GradeMark requires students to submit directly to this system, requiring interaction with a system different from the LMS. 286 Australasian Journal of Educational Technology, 2012, 28(2) Markers Assistant [http://www.surreal.com.au/] also has a rubric and facilitates the work of multiple markers (Wells, 2006). Systems like GradeMark and Markers Assistant have disadvantages like needing external interfacing with course and student lists, or requiring students to access a system separate from their LMS. In contrast to looking at generic software and solutions separate of LMS, it is easy to find arguments for creating a solution for assignment management and marking based on LMS. The literature emphasises that assessment is an integral part of teaching and learning. Therefore, assessment support should take place within the environment that is used for supporting teaching and learning. Assessment in tertiary education is bound firmly into course structures. Assessment is scheduled as part of a course and is undertaken by the students enrolled in the course. LMS already capture course structures and student enrolments. They regulate student and teacher access. LMS already provide some support for assignment management and marking. It therefore makes sense to link assignment support into the existing structures of learning management systems. The next section takes a closer look at the assignment features of two prominent LMS. Comparison of LMS capabilities against requirements Current LMS already provide some assistance for the management and marking of assignments. They largely support what has been described as pre- and post-marking tasks in Figure 3. The pre-marking tasks are comprised of setting up the assignment and making the task specification available to students, and of handling the submission of student work. The post-marking tasks include the release of marking results to students and the bookkeeping of grades. While aspects of the support for pre- and post-marking tasks can be improved, the tasks are covered in principle by today’s LMS. The big gap in LMS provision of assignment support lies at the core of the marking and quality assurance tasks. As the education literature emphasises, marking rubrics are central to high quality marking and feedback. Current LMS either have no or only limited concepts of marking rubrics. Further, marker allocation, meaning knowing who is assigned to mark whom, is central to both progress checking and quality monitoring. There is currently no implementation of marker allocations in prominent LMS. This means that while there is reasonable support for the handling of student work and marking results, there is only very limited support for the actual marking. The two LMS Moodle [http://moodle.org/] and Blackboard [http://blackboard.com/], are currently the most prominent LMS at tertiary institutions in Australia and New Zealand. Table 1 takes a closer look the assignment related functionalities of Moodle and Blackboard by comparing these functionalities against the tasks identified in Figure 2. The information to evaluate the systems has been taken from the descriptions of features of the systems on their websites, from conversations with system representatives at conferences and from using the systems. The examinations of Moodle and Blackboard suggest that these systems provide base functionality for assignment management and marking. They largely handle the pre- and post-marking tasks. The shortcomings of both LMS are the lack of appropriate support for high quality and timely marking. Up to version 2.2 (released late 2011) the Moodle assignment module did not contain rubrics at all (the add-on Workshop module Heinrich, Milne and Granshaw 287 Table 1: LMS functionality compared to requirements for the marking and management of assignments Task Description Moodlehttp://moodle.org/ Blackboard http://blackboard.com/ Setting up the assignment Creating assignment drop box; specifying parameters like due date and submission options; making task specification available. Provided. Provided. Handling submissions of student work Storing files students have submitted; recording submission parameters like date and time. Provided. Provided. Marking preparation Creating marking rubric including comment banks; allocation of markers to students. Some support for rubrics (since Moodle 2.2). No support for marker allocation. Basic rubric support (more with Turnitin plugin). No support for marker allocation. Marking Reading student work. Producing detailed feedback to students aligned to criteria specified in the marking rubric. Deciding on marks or grades aligned to criteria specified in the marking rubric. Student work downloaded and opened in appropriate application. Text field for feedback (limited rubric support since 2.2). Single mark or grade (limited rubric support since 2.2). Student work downloaded and opened in appropriate application. Text field for feedback (rubric can only be looked at by instructor; criteria are not transparent to students). Single mark or grade (no details per rubric criterion). Monitoring progress Checking on progress of marking to facilitate timely return to students. Only if markers put feedback/marks into Moodle, where it is visible to students (if assignment settings are not changed to prevent this); no specific support. Only if markers put feedback/marks into Blackboard, where it is visible to students (if assignment settings are not changed to prevent this); no specific support. Quality control Reviewing the quality of individual marking; looking at inter- and intra- marker consistency. Only if markers put feedback/marks into Moodle, where it is visible to students (if assignment settings are not changed to prevent this); no specific support. Only if markers put feedback/marks into Blackboard, where it is visible to students (if assignment settings are not changed to prevent this); no specific support. Release to students Making the marking result comprised of marks and feedback available to students. Provided, but upload of feedback sheets or annotated student work requires manual upload for each student. Provided, but upload of feedback sheets or annotated student work requires manual upload for each student. Bookkeeping of marks Retaining assignment marks as they will contribute to overall course grades. Provided. Provided. Reflection on assignment Reflecting on task specification, and marking in light of desired learning outcomes. No specific support. No specific support. 288 Australasian Journal of Educational Technology, 2012, 28(2) contains rubrics but serves a purpose different from assignments). Moodle 2.2 contains a rubric based on quality levels and criteria, yet lacks additional features such as descriptions for markers or comment banks. Blackboard allows an instructor to create a rubric. Yet, the rubric features are severely limited. During marking a rubric can only be looked at by the marker. There is no way to give comments or marks linked to the rubric criteria. Students do not see the rubric at all, neither in preparing their work nor in understanding the feedback and marks they are given. The rubrics do not contain comment banks and do not allow providing instructions to markers. If the Turnitin plug-ins Blackboard Basic or Blackboard Direct [http://submit.ac.uk/en_us/support/ integrations/blackboard-basic] are integrated, Blackboard users have access to stronger support via rubrics, but still lack important features. Neither Moodle nor Blackboard support marker allocations. Therefore, progress checking aiming at timely return of marking to students is limited in both systems. Similarly, there is no support for inter-marker consistency checking. Without full support for marking rubrics and marker allocations, Moodle and Blackboard have fundamental shortcomings for assignment marking. Yet, based on their prominence in other areas of support for teaching and learning and their pre- and post-marking features for assignment management, the systems are worth building on. The next section looks at two possible ways to do so. Pure LMS versus LMS + application solutions Several factors need to be considered when deciding between creating a pure LMS solution or a combination of LMS and a specialised marking application. Assignment marking occurs in a wide variety of contexts. As a consequence, student work will be submitted in a wide range of formats and file types. While many assignments will be word processing documents, other submissions could be spreadsheets, presentation files, computer programs or compressed file collections. This variety of formats means that these files have to be downloaded on the teacher’s desktop computer, as they cannot be executed within a web-browser. Direct annotation of student files is common for providing context specific inline comments. Across a variety of file formats and without changing the layout of the student work, this again requires files to be stored on the desktop computer. Dealing with file storage and management works much better on the desktop than in a fully web-based solution where security restrictions intervene. Many teachers and their markers still want, and at times require, the possibility to mark offline, without the continuous presence of a network connection. With a combined system architecture, users can download all marking information when they are in reach of a network connection. They can then mark offline using the specialised application and resynchronise once the network connection is again available. A fully web-based solution would require continuous network connection. Despite advances in web-application technologies, desktop solutions still provide the developers with more control over the user interface of the application. A desktop application can therefore provide a better user experience, which is important when dealing with the complex processes and data required for assignment assessment. Heinrich, Milne and Granshaw 289 There are downsides to implementing a desktop solution, such as the requirement for every user to install the specialised application on their desktop. Desktop and LMS versions have to be compatible in version numbers. While installer files and automated update scripts can support application installation and updates, this still implies an overhead for users and the potential for problems due to version conflicts. Being able to work offline requires local data storage until the next synchronisation with the LMS site is possible. Local databases and file storage on the desktop client need to be used in the meantime. Local data storage introduces a potential risk to data safety and security. Multiple members of a marking team will work on the same overall assignment from multiple desktops. The application therefore needs to guard against inconsistencies between data on synchronisation with the LMS. On balance, consideration of these technical issues suggests that a thoughtful implementation based on a combination of specialised application and LMS is more promising than a pure LMS implementation. Following these considerations on system architecture, a concrete e-learning solution for assignment management and marking is introduced in the next section. Architecture and implementation of LMS + application solution The following provides a brief overview of the architecture and implementation of a solution combining a specialised application with functionality already provided in a LMS. The specific systems discussed are Lightwork [http://lightworkmarking.org] and Moodle. Lightwork has been implemented as a desktop application that interfaces closely with Moodle. The desktop application has been written in the Java programming language and runs on Windows, MacOS and Linux platforms. To enable the desktop application to interface with a Moodle site, the Moodle site needs to be extended (from Moodle 2.3 or 2.4 onwards these extension will be part of the Moodle core). In particular, this means adding web-service code to the PHP code of Moodle for the communication between Lightwork clients and the Moodle site. Lightwork specific tables need to be added to the Moodle site’s database. This allows Lightwork to store data related to marking rubrics, marker allocation and marking feedback. A mapping needs to be established between Moodle roles and Lightwork capabilities. Lightwork follows the requirements of close integration with Moodle and of taking advantage of functionality already implemented in standard Moodle. Teachers use Moodle functionality to set up assignments. Students then use the normal Moodle user interface to submit their work and view their marking results. Once released by the teacher to students, the marking results are captured in Moodle, in the same way as if marking had occurred directly in Moodle. The important concepts Lightwork adds to the marking process are marking rubrics and marker allocation, resulting in improvements to marking efficiency and quality. Figure 3 provides an overview of the assignment related interaction of students with Moodle and of teachers with Moodle and Lightwork. Figure 4 shows a screenshot of Lightwork. The direct interactions of teachers and students with Moodle can only occur when a network connection is present. In contrast, a network connection for working with Lightwork is only required for the periods of data exchange between Lightwork and Moodle, for example for the download of student assignment submissions. The actual marking and management tasks can be carried out in Lightwork in an offline mode, without network connection. 290 Australasian Journal of Educational Technology, 2012, 28(2) Figure 3: Overview of the interaction of students with Moodle and of teachers with Moodle and Lightwork Figure 4: Screenshot of the marking interface of Lightwork (showing fictitious data) Heinrich, Milne and Granshaw 291 Conclusions This article lays out the tasks involved in assignment management and marking. The specification of requirements for an e-learning solution is derived from these tasks. Figure 5 summarises the various architectures for an e-learning solution considered in this article. It is suggested that the pathway to a successful solution needs to incorporate the LMS. The combination of LMS and a specialised application has been implemented with Moodle and Lightwork and has been available for use in real teaching settings from mid 2010. Early evaluations with teachers from four tertiary institutions in New Zealand indicate a positive effect on the efficiency and quality of assignment marking (Heinrich & Milne, 2012). To the best knowledge of the authors at the time of writing this article, none of the current widely used LMS comes close to satisfying all requirements for assignment management and marking. Figure 5: Evaluation of pathways Yet, LMS development is moving on and new functionalities are being incorporated. For example, for Moodle work is underway to incorporate key Lightwork features such Architectures for e-learning solutions for assignment management and marking LMS-based solutions Generic solutions or specialised solutions independent of LMS Application + LMS Pure LMS Successful implementation with Moodle + Lightwork – Recommendation to follow this pathway Dead end – Pathway not recommended No full implementation available yet – Can this pathway provide full functionality? 292 Australasian Journal of Educational Technology, 2012, 28(2) as comment banks, marker allocation and marking statuses into the core [see http://docs.moodle.org/dev/Lightwork]. This work, together with other improve- ments planned for the assignment activity [see http://docs.moodle.org/dev/ Assignment] will provide much better support for assignment marking purely based on the features integrated into the LMS. New technologies, namely HTML5 [see http://www.w3.org/TR/html5/], bring local data storage to a browsing interface, opening possibilities for offline work without requiring a separate application. While the current Moodle developments, scheduled for release mid to late 2012, will bring big improvements, they will still not have the full functionality around offline work and non text-based file formats that a separate application like Lightwork can offer. It will be interesting to observe if the user community will value the additional functionality enough to carry the overheads required in supporting and maintaining a separate application. This article has introduced requirements for assignment management and marking based on pedagogy and practical realities of tertiary education. From there, the considerations for leading from requirements to potential architectures have largely been made on technical issues. To conclude the article it seems appropriate to return the attention to the core purposes of LMS in supporting teaching and learning. LMS are very powerful and increasingly complex systems that bring learners and teachers together. LMS provide a protected course environment in which they facilitate the exchange of documents, information and thoughts to facilitate learning. Yet, LMS are largely not the right place for cognitively complex work. For example, teachers do not construct their course material like study guides or lecture slides inside the LMS. Instead, they prepare such material outside the LMS and only use the LMS to facilitate access. Similarly, students do not construct their assignment work inside the LMS. They construct outside and only use the LMS for submission. Following the same pattern an argument can be made for assignment marking. The cognitively complex tasks around assignment marking, that is construction of the marking rubric and marking of student work, should not take place inside the LMS. There seems to be a current tendency towards overloading LMS by forcing work of increasing cognitive complexity into these systems. A careful reconsideration of the purpose of LMS may point towards comprehensive solutions that complement LMS rather than overloading them and thus maintaining cognitive fit. References Brown, G. (2009). The reliability of essay scores: The necessity of rubrics and moderation. In L. Meyer, S. Davidson, H. Anderson, R. Fletcher, P. Johnston & M. Rees (Eds.), Tertiary assessment & higher education student outcomes: Policy, practice & research. Wellington, New Zealand: Ako Aotearoa, pp 43-50. http://akoaotearoa.ac.nz/ako-aotearoa/ako- aotearoa/resources/pages/tertiary-assessment-higher-education-student-outcomes-poli Burrows, S. & Shortis, M. (2011). An evaluation of semi-automated, collaborative marking and feedback systems: Academic staff perspectives. Australasian Journal of Educational Technology, 27(7), 1135-1154. http://www.ascilite.org.au/ajet/ajet27/burrows.html Denton, P., Madden, J., Roberts, M. & Rowe, P. (2008). Students’ response to traditional and computer-assisted formative feedback: A comparative case study. British Journal of Educational Technology, 39(3), 486-500. http://dx.doi.org/10.1111/j.1467-8535.2007.00745.x Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203-214. http://dx.doi.org/10.1111/j.1467-8535.2008.00915.x Heinrich, Milne and Granshaw 293 Gronlund, N. E. (2006). Assessment of student achievement. Boston: Pearson. Hanna, G. S. & Dettmer, P. A. (2004). Assessment for effective teaching using context-adaptive planning. New York: Pearson. Hattie, J. (2009). The black box of tertiary assessment: An impending revolution. In L. Meyer, S. Davidson, H. Anderson, R. Fletcher, P. Johnston & M. Rees (Eds.), Tertiary assessment & higher education student outcomes: Policy, practice & research. Wellington, New Zealand: Ako Aotearoa, pp 259-275. http://akoaotearoa.ac.nz/ako-aotearoa/ako-aotearoa/resources/ pages/tertiary-assessment-higher-education-student-outcomes-poli Heinrich, E. & Milne, J. (2012). Applying a framework to evaluate assignment marking software: A case study on Lightwork. Research in Learning Technology, 20(2). http://researchinlearningtechnology.net/index.php/rlt/issue/view/1432 Heinrich, E., Milne, J. D. & Moore, M. (2009). An investigation into e-tool use for formative assignment assessment - status and recommendations. Educational Technology & Society, 12(4), 176-192. http://www.ifets.info/journals/12_4/16.pdf Heinrich, E., Milne, J. D., Ramsey, A. & Morrison, D. (2009). Recommendations for the use of e- tools for improvements around assignment marking quality. Assessment and Evaluation in Higher Education, 34(4), 469-479. http://dx.doi.org/10.1080/02602930802071122 Heinrich, E., Milne, J., Crooks, T., Granshaw, B. & Moore, M. (2007). Literature review on the use of e-learning tools for formative essay-type assessment. [verified 12 Mar 2012] http://etools.massey.ac.nz/research.htm Kendle, A. & Northcote, M. (2000). The struggle for balance in the use of quantitative and qualitative online assessment tasks. In Learning to choose. Choosing to learn. Proceedings ASCILITE Coffs Harbour 2000. http://www.ascilite.org.au/conferences/coffs00/papers/amanda_kendle.pdf Lambert, D. & Lines, D. (2000). Understanding assessment: Purposes, perceptions, practice. Padstow: TJ International. Linn, R. L. & Miller, M. D. (2005). Measurement and assessment in teaching. Columbus: Pearson Merrill Prentice Hall. Macdonald, J. (2003). Assessing online collaborative learning: Process and product. Computers & Education, 40(4), 377-391. http://dx.doi.org/10.1016/S0360-1315(02)00168-9 Margrain, V., Everiss, L., Murphy, T., Edlin, A., McClew, J. & Meade, A. (2009). “Ka pai – well done”: Student teacher perceptions of assessment feedback in distance learning. In L. Meyer, S. Davidson, H. Anderson, R. Fletcher, P. Johnston & M. Rees (Eds.), Tertiary assessment & higher education student outcomes: Policy, practice & research. Wellington, New Zealand: Ako Aotearoa, pp 129-139. http://akoaotearoa.ac.nz/ako-aotearoa/ako-aotearoa/ resources/pages/tertiary-assessment-higher-education-student-outcomes-poli Milne, J., Heinrich, E., Ramsay, A., Morrison, D., Granshaw, B. & Moore, M. (2007). Survey report on the use of e-learning tools for formative essay-type assessment. [verified 12 Mar 2012] http://etools.massey.ac.nz/research.htm Moore, M., Granshaw, B., Milne, J. & Heinrich, E. (2007). Software tools which can support working with essay-type assignments. [verified 12 Mar 2012] http://etools.massey.ac.nz/research.htm Muirhead, B. (2002). Effective online assessment strategies for today's colleges & universities. Educational Technology & Society, 5(4). http://www.ifets.info/journals/5_4/discuss_summary_october2002.pdf 294 Australasian Journal of Educational Technology, 2012, 28(2) Nitko, A. J. (2004). Educational assessment of students (4th ed.). Upper Saddle River, New Jersey: Pearson Education. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. http://dx.doi.org/10.1007/BF00117714 Sadler, D. R. (2009). Indeterminacy in the use of preset criteria for assessment and grading. Assessment & Evaluation in Higher Education, 34(2), 159-179. http://dx.doi.org/10.1080/02602930801956059 Wells, J. (2006). Markers Assistant: A software solution for the management of the assessment process. International Journal on ELearning, 5(3), 439-458. http://www.highbeam.com/doc/1P3-1066521781.html Whitelock, D. (2009). Editorial: e-assessment: Developing new dialogues for the digital age. British Journal of Educational Technology, 40(2), 199-202. http://dx.doi.org/10.1111/j.1467- 8535.2008.00932.x Whitelock, D. & Brasher, A. (2006). Developing a roadmap for e-assessment: Which way now? In M. Danson (Ed), Proceedings of the 10th CAA International Computer Assisted Assessment Conference. Loughborough, UK: Professional Development, Loughborough University, pp. 487-501. Whitelock, D. & Watt, S. (2007). E-assessment: How can we support tutors with their marking of electronically submitted assignments? Ad-Lib Journal for Continuing Liberal Adult Education, 32, 7-9. http://oro.open.ac.uk/12010/1/Whitelock_e-Assessment_paper.pdf Authors: Dr Eva Heinrich, Senior Lecturer in Computer Science School of Engineering and Advanced Technology, Massey University Private Bag 11 222, Palmerston North, New Zealand Email: e.heinrich@massey.ac.nz Web: http://seat.massey.ac.nz/heinrich http://lightworkmarking.org/ Lightwork - Managing Marking Effectively Mr John Milne, Teaching and Online Consultant Wellington Centre for Teaching and Learning, Massey University Email: j.d.milne@massey.ac.nz Mr Bruce Granshaw, Lecturer, School of Education Policy and Implementation Victoria University of Wellington. Email: bruce.granshaw@vuw.ac.nz Please cite as: Heinrich, E., Milne, J. & Granshaw, B. (2012). Pathways for improving support for the electronic management and marking of assignments. Australasian Journal of Educational Technology, 28(2), 279-294. http://www.ascilite.org.au/ajet/ajet28/heinrich.html