Conference Reflections Compass: Journal of Learning and Teaching, Vol 11, No 1, 2018 Using technology as a way of evolving assessment practice Sandra Clarke University of Greenwich A 30-credit, level 5 land law course is delivered to around ninety students per year in the University of Greenwich, a post-1992 UK University. The original assessment regime consisted of five items of assessment: Figure 1: The problem Although each of these assessments had a sound justification individually, some small- stakes assessments were either too stressful (the effort required to produce and upload the web page/online poster accessed only 5% of the total marks for the course) or tripped up even diligent students (though all students uploaded the learning log weekly, some of them forgot the final hand-in). The timed assignment, a seen examination sat in January each year, was a very good formative and summative assessment, but generated piles of long, hand-written scripts which were very difficult to mark and return within the fifteen working days required by the Assessment and Feedback Policy of the University of Greenwich. Such considerations, together with proposed changes to professional legal education, made it a sensible time to consider alternative methods of assessment – those using learning technology. Key drivers for change were: Figure 2: Key drivers for change Reduction of marking load Reduction of student stress Avoidance of traps for the unwary Preparation for changes to Legal Education – the SQE But… still keeping the benefits of the former assessments Conference Reflections Compass: Journal of Learning and Teaching, Vol 11, No 1, 2018 Decisions made fell broadly into three categories: Figure 3: Change decisions This article will concentrate on two items of assessment which it was decided could be assessed in a different but equally effective way by utilising learning technology already available within Moodle. 1. The web page/poster assessment The web page/poster assessment has been previously described by the author (Clarke et al., 2010). Briefly, the task requires the student to choose and photograph any object (a chair, a bridge, a statue) and to apply the law to determine whether that object forms part of land or remains a separate chattel, using the authorities studied in the early part of the course. To alleviate the marking load, a rubric was set up in TurnitinUK for the swift grading of submitted posters. However, students suffered stress whenever technical problems prevented their assignments from being uploaded or displayed correctly. In addition, since TurnitinUK’s anonymous marking function renders document-embedded links unclickable, references had to replace links, thus diminishing the value of the electronic format. It was therefore decided that the poster should instead be a formative, peer-reviewed and peer-marked exercise, using the Moodle Workshop tool. The students uploaded their assignments, which were then assigned to five other students for online marking – on personal electronic devices – within seminar time, using the rubric provided and allowing for free-text final comments. Prior to this peer-marking exercise, the lecturer led class discussion about assessment and feedback on the basis of the REAP principles (REAP project, 2010). The exercise itself was designed in accordance with ‘Seven good practices for undergraduate education’ (Chickering and Gamson,1987): it encouraged in- and out-of- class contact between students and lecturers, developed reciprocity and cooperation among students, used active learning techniques, gave prompt feedback (within forty-eight hours of hand-in), emphasised time on task and respected diverse talents and ways of learning (particularly as it was a creative exercise, unusual in a law degree). Students also expressed amazement at and respect for the best submissions, which communicated high expectations for the course. There were some problems: the Moodle workshop tool is quite complicated for the lecturer to set up correctly, so careful attention to detail is required throughout; not all students participated (they failed either to upload a poster or to attend the assessment seminar); students using phones as marking devices found the rubric difficult to access; the lecturer had to mark some posters whose assigned students had not done so; a small number of posters did not upload correctly (usually on account of student inability to follow advice about acceptable file formats) to the disappointment of the students concerned. Improvements in response to these initial challenges will, next year, include: an in-class opportunity for the lecturer to discuss each poster with its author and provide oral feedback; an invitation to drop • Learning log change • Web page/poster • January exam keep • Research coursework • Final exam Conference Reflections Compass: Journal of Learning and Teaching, Vol 11, No 1, 2018 students to email to the lecturer any posters which fail to display correctly, so that feedback may still be acquired. Nevertheless, in spite of such difficulties, this was a largely successful change, retaining the learning outcomes of the original assessment while reducing stress and adding the chance of experiencing peer marking. 2. The January exam The second change was from the seen written exam in January to a seen online assessment. Both assessments test all the material taught in the first term, including the topic covered in the formative poster assignment. The assignment consists of twenty questions, a mix of multiple choice (MCQs), extended matching and ordering questions, created using the Moodle quiz tool. They are based upon a register of title document, which is available to students before the Christmas break, as are the questions asked (but not the answer options). Students can seek guidance about the assessment on a dedicated Moodle forum. Students take the assessment unsupervised during a period of eight hours. Once they begin, they are timed and cannot restart. Questions and options within questions are shuffled randomly to make collusion more difficult – although it is not impossible. An improvement would be for the assignment to be taken under controlled conditions. Professional online examinations in law, such as the Qualified Lawyers Transfer scheme, allow just under two minutes per question, so forty minutes were allowed for the test. The Moodle quiz tool – which allows extra time to be added for students, with reasonable adjustments for specific learning needs – proved straightforward to use. It is, however, important to create the questions in named question banks, before producing a quiz and adding questions to it. This facilitates the re-use of questions in later quizzes. Research is available on the use of online testing in disciplines where understanding and problem-solving are required skills rather than, for example, accurate calculation (Fry et al., 2013). Several benefits have been identified, including: the ability to test a number of different topics within a shorter exam; the avoidance of the need for handwriting under pressure when students are more used to typing; automatic, unbiased marking which is easy and quick, no matter how large the cohort. There are also perceived drawbacks – students might pass by guessing the answer or working it out by spotting the relevant words without understanding the concepts (Deane and Bozin, 2017). Online exams also fail to test writing skills, which are essential to lawyers. These concerns must be answered if the benefits of this type of assessment are to outweigh the difficulties. The answer lies, firstly, in using online assessment as one of a number of assessments on the course (writing skills are tested in the unchanged research coursework and end-of-year exam, for example). Secondly, problems such as guessing or recognising the answer without understanding it are best solved by appropriate question design, which is a significant challenge. MCQs, which form the majority of the questions, consist of three main parts – the stem, the lead in and the options, which latter consist of the key (best answer) and a number of distractors (less good / less correct answers). The stem of a MCQ needs to be clear and avoid unnecessary detail. It should present a realistic, meaningful scenario which avoids ambiguity. It should test a key rule, not ‘nitpicky facts’ (Case and Donahue, 2008). Conference Reflections Compass: Journal of Learning and Teaching, Vol 11, No 1, 2018 The lead-in should ideally be able to be answered by the student before the options are seen. A student who knows the legal rule should be able to apply it and reach an answer. Lead-ins to avoid include ‘Which of the following is correct?’ and, even worse, ‘Which of the following is incorrect?’ The options should contain a clear key and a number of distractors which should be plausible and not too different in length and style from the key. Using correct technical vocabulary in the distractors minimises the likelihood of success by ‘recognising without understanding’. These requirements make setting questions difficult and time-consuming. However, the work can be spread over a longer period, rather than compressed into the fifteen-day marking limit. It is essential to involve colleagues in a thorough testing process. However, it is possible to exclude one question from the overall grade if it is later realised that it was unfair or unreliable. Moodle produces analytics which can be used to assess performance across the test as whole and within particular questions. Overall, the changes to assessment in the course were successful. The students enjoyed the poster assessment and the online exam and appeared to work hard for both assignments. The results in the course were in line with those in previous years (mean grade 54% in 2015-16 and 56% in 2016-17) and there was good student satisfaction. There was no reduction in workload for the teaching team, but the work could be moved to less busy times of year, which made it less stressful. Lessons learnt from this first implementation of the changes will serve to refine future iterations. Reference list Case S. and Donahue E. (2008) ‘Developing High-Quality Multiple-Choice Questions for Assessment in Legal Education.’ Journal of Legal Education, Volume 58, Number 3 available at http://www.law.smu.edu/Media/Faculty/Faculty%20Teaching%20Resources/Assessing%20S tudent%20Work/Developing-High-Quality-Multiple-Choice-Questions.pdf (Accessed 12 March 2018). Chickering, A.W. and Gamson, Z.F. (1987) Seven Principles for Good Practice in Undergraduate Education. AAHE Bulletin, March. Available at: http://www.lonestar.edu/multimedia/sevenprinciples.pdf (Accessed: 30 October 2017). Clarke S., Crofts, S., Laycock A. and Phillips E. (2010) ‘Exceeding the Boundaries of Formulaic Assessment: Innovation and Creativity in the Law School.’ The Law Teacher 44(3), 334-364. Deane, F. and Bozin, D. (2017) ‘Using Guiding Principles to Construct Effective Multiple Choice Exams to Assess Legal Reasoning.’ Legal Education Review: 26(1), Article 1. Available at: http://epublications.bond.edu.au/ler/vol26/iss1/1 (Accessed: 30 October 2017). Fry, E., Crewe, J. and Wakeford, R. (2013) ‘Using multiple choice questions to examine the content of the qualifying law degree accurately and reliably: the experience of the Qualified Lawyers Transfer Scheme.’ The Law Teacher, 47(2), 234-242, DOI: 10.1080/03069400.2013.790156 Available at: http://www.law.smu.edu/Media/Faculty/Faculty%20Teaching%20Resources/Assessing%20Student%20Work/Developing-High-Quality-Multiple-Choice-Questions.pdf http://www.law.smu.edu/Media/Faculty/Faculty%20Teaching%20Resources/Assessing%20Student%20Work/Developing-High-Quality-Multiple-Choice-Questions.pdf http://www.lonestar.edu/multimedia/sevenprinciples.pdf http://epublications.bond.edu.au/ler/vol26/iss1/1 Conference Reflections Compass: Journal of Learning and Teaching, Vol 11, No 1, 2018 https://www.tandfonline.com/doi/abs/10.1080/03069400.2013.790156 (Accessed 12 March 2018). REAP project. (2010) Available at: https://www.reap.ac.uk/Home.aspx (Accessed: 30 October 2017) Helpful Resources Moodle Docs. Available at: https://docs.moodle.org/32/en/Workshop_quick_guide (Accessed: 30 October 2017). Moodle Docs. Available at: https://docs.moodle.org/34/en/Quiz_statistics_report (Accessed: 30 October 2017). https://www.tandfonline.com/doi/abs/10.1080/03069400.2013.790156 https://www.reap.ac.uk/Home.aspx https://docs.moodle.org/32/en/Workshop_quick_guide https://docs.moodle.org/34/en/Quiz_statistics_report