Microsoft Word - blayney.doc Australasian Journal of Educational Technology 2004, 20(2), 209-231 Automated formative feedback and summative assessment using individualised spreadsheet assignments Paul Blayney and Mark Freeman University of Sydney This paper reports on the effects of automating formative feedback at the student’s discretion and automating summative assessment with individualised spreadsheet assignments. Quality learning outcomes are achieved when students adopt deep approaches to learning (Ramsden, 2003). Learning environments designed to align assessment to learning objectives and learning activities encourage these approaches (Biggs 1999). A crucial part of any learning or assessment activity is the degree to which students receive timely and effective feedback. As academics have experienced more pressure, frequently feedback has been limited to a single score, achieved most commonly by auto-corrected multiple choice questions in ICT supported environments. Spreadsheet assignments are a good way to learn and demonstrate understanding of concepts requiring calculation and interaction of different elements. However, they can be an assessment nightmare either because of validity problems (the potential for cheating using the cell copy function) or because of marking time (if students are allowed individual choice in application topic). This paper responds to Higgins, Hartley and Skelton's (2002) observation on the lack of research on feedback and builds on work by Lehman and Herring (2003) in using interactive spreadsheets to provide immediate feedback by describing effects on students. Effects on academics and their productivity are considered, including Rogers’ (1995) diffusion of innovation factors. Academics and academic managers seeking ways to improve learning by improving feedback, without an increasing workload, will find this research of interest. 1. Introduction Quality learning outcomes are achieved when students adopt deep approaches to learning (Ramsden, 2003). Academics can encourage deep approaches by designing learning environments that align assessment to learning objectives and learning activities (Biggs 1999). A crucial part of any learning or assessment activity is the degree to which students receive 210 Australasian Journal of Educational Technology, 2004, 20(2) timely and effective feedback. The limited evidence on feedback suggests that students want more (Higgins, Hartley and Skelton, 2002). In the last decade there has been an increasing interest in harnessing technology to deliver learning and assessment activities that simultaneously assist students’ learning and improve academic productivity in higher education in the face of ever increasing class sizes and diminishing resources. The objective of this paper is to report the effects of a spreadsheet assignment method that allows students to self assess and subsequently allows staff automatic summative assessment upon file submission. The method goes beyond Lehman and Herring (2003) who describe an alternative spreadsheet method for providing immediate formative feedback and the resultant benefits to student learning, in that our method systematically discourages cheating. Furthermore, it also increases academic productivity since the automation can be applied to any class size. Section 2 reviews the relevant literature. Section 3 describes the context and process for developing this innovative use of spreadsheet software. Section 4 describes the research method, section 5 the results and some discussion, with concluding remarks provided in section 6. 2. Relevant literature That assessment is crucial to learning is well known (Ramsden, 2003). Further, Biggs (1999) strongly argues that alignment of assessment with learning objectives, resources and activities is a key strategy to ensure quality outcomes. However, following Prosser and Trigwell’s (1999) results, it seems that some academics view teaching as helping students change their conceptions rather than as simply transmitting information to them. These educators are more likely to design and support subjects that encourage students to take a deep approach to learning. Feedback is a worthy focus of academic effort since it focuses students on what they need to do to improve. Whether verbalised, or provided by the academic’s choice of textbook with end of text answers and model solutions, academics have attempted to help students assess their progress and move forward in their learning. In the current resource constrained higher education environment, academics are increasingly considering learning and assessment solutions that utilise information and communications technology (ICT). In recent years there has been increased interest in ICT that promises wide adoption possibilities and economies of scale. However, there appears ample evidence, such as Russell’s (1999) review of 355 research reports, that ICT in itself is no guarantee of improved learning. Any improvements in Blayney and Freeman 211 learning outcomes depend on how ICT is used. For example, in their national review of educational innovation in Australian higher education, Alexander and McKenzie (1998) show that innovative ICT introduced without changes to assessment was less likely to realise the expected improvement in learning outcomes. Thus we would expect successful academics to make explicit decisions about feedback and ICT, two important elements in assessment, if they are to encourage students to take a deep approach to learning. The literature is not expansive in regard to feedback (Smith and Coombe, 2000) although there are notable recent exceptions such as Higgins, Hartley and Skelton (2002), who posit that larger class sizes have meant the demise of written feedback to students. Despite student expectations of feedback as a right, they observe that feedback received was minimal and, in 40% of cases, the handwriting was hard to read. As a result, students rarely spent more than 10-15 minutes reading and reflecting on the feedback comments provided. Feedback on student work presents a major dilemma to higher education since students want more of it but academics are increasingly subject to time constraints. As staff-student ratios have increased, sufficient and timely feedback comes at an increasingly higher cost to academics. Outsourcing marking to sessional academic staff has some attraction, but Smith and Coombe (2000) note some serious flaws. ICT can help address this dilemma. One ICT solution that provides immediate formative feedback in a classroom is facilitated by small piece of hardware called a personal response system (eg. Varitronix and eInstruction). Elliot (2003) finds very positive results in using a personal response system in teaching microeconomics. She noted “student interest and concentration, enhancing active learning and the level of interaction in a lecture setting, whilst allowing students as well as lecturers an opportunity to monitor the level of student understanding”. Epstein, Epstein and Brosvic (2001) report improved knowledge retention from classroom multiple choice tests where immediate feedback was provided using portable scanning technology (e.g. Scantron). Convergence of wireless technology and personal digital assistant (PDA) devices will be another option for instant feedback in classroom contexts in the near future as wireless PDAs become more affordable. Outside the classroom there are a range of potential ICT solutions to support feedback. The simplest option, analogous to end of text resources, includes providing generalised feedback or model answers through a course website to allow students to self select and apply feedback themselves. However, more automated and individualised options are available. Historically, ICT supported assessment was restricted to scanning multiple choice tests, with feedback provided after some delay 212 Australasian Journal of Educational Technology, 2004, 20(2) for the scanning process. In recent years the proliferation of standardised course management systems (e.g. Blackboard and WebCT) or those that focus specifically on assessment (e.g. QuestionMark and Perception) have increased the potential impact of ICT. Academics can now use the quiz feature on these systems to provide assessment in an online environment since they do not require academics to learn HTML code. Although academic productivity improves, the gain can come at a cost. For example, multiple choice questions can be difficult and time consuming to write well, especially for testing higher learning outcomes like analytical and problem solving skills, and feedback may be sacrificed with time constraints. In addition, multiple choice questions cannot allow students to demonstrate important graduate attributes such as the ability to collaborate and communicate. Even after considerable design effort, biases may still be inherent such as those due to language - students assessed in a language other than their native tongue are susceptible to nuances in language which can be difficult to identify. Artificial intelligence systems, developed to assess free response answers on open ended questions appear attractive - one vendor claiming that “IntelliMetric achieves levels of scoring accuracy that equal or exceed expert graders” (Vantage, 2003) – but the expense of complex, proprietary heuristic systems is likely to place it beyond the reach of the average academic. Other ICT supported solutions can be used in conjunction with human markers to promote learning through improved feedback. Mindtrail and MarkIt are two programs that encourage academics to generate knowledge trees related to marking criteria and from which standardised feedback comments can be chosen. Further customisation of the feedback for each student’s assignment is possible with such programs. However, the feedback provided is neither fully automated, since it requires intervention by a human marker in selecting the appropriate feedback, nor is it immediate, since such human marker intervention takes time. Because student attention is more critically focussed by assessment, we expect failure of ICT supported assessment innovations to be of more concern to students than failure in ICT supported learning innovations. Failures and inaccuracies could arise from students (eg. data entry errors), academics (eg. assessment tasks that preclude different possible student approaches), markers (eg. applying ambiguous or poorly expressed assessment criteria) or ICT itself (.g. Internet connections dropping out; incorrect answers being programmed as correct; or simply corrupt files). In all, we can identify nine different parts of the assessment cycle in which ICT can assist. Blayney and Freeman 213 1. Assigning assessment tasks to students 2. Clarifying assessment requirements for students 3. Students completing part or all of an assessment task 4. Delivering formative feedback on an assessment 5. Students submitting their assessment task 6. Marking student work submitted for assessment 7. Generating feedback on assessments for students 8. Delivering feedback on summative assessments to students 9. Evaluating effectiveness of assessment tasks But what stops the adoption of innovation? The classic work on diffusion of innovation by Rogers (1995) argues that dissemination into the mainstream requires potential adopters seeing the relative advantage over current practice, can ‘try before they buy’, has visibly positive results, is not too complex and is compatible with current practices and values. On the importance of compatibility, Chickering and Ehrmann (1996) were the first to note the case for promoting w o r l d w a r e in higher education. Applications such as spreadsheet programs, word processors and web browsers are already widely available and used outside higher education. This research project focuses on the use of spreadsheets and the effects on students and staff when adopted into the learning and assessment process. Spreadsheets work best and can be applied to any discipline that has rule based problems (eg. finance, tax, accounting and physics) and such calculations and interactions between underlying variables are crucial knowledge for professionals. While there are numerous examples using spreadsheets in higher education, there is limited evidence of their effects on staff and students in their use described in points 4, 6 and 7 above, namely automating the delivery of formative self assessment feedback as well as to mark and generate summative assessment feedback where spreadsheets are individualised. Since their early applications in teaching (eg. Izard and Reeve, 1986), traditional spreadsheet assignments have been regarded as a good way for students to learn concepts. Their limitations have also been noted particularly when used for assessment. Although Edmonds and Tsay (1989) show how spreadsheet macros can be used to automate summative assessment (described in point 6 above), validity problems still exist because students have the potential for cheating by copying another’s file or by using the cell copy function. Alternatively, if students are allowed choice in developing their own spreadsheet solution to avoid such validity problems, or spreadsheets are used to generate individualised assessment problems by merging with word processing programs (Teets and Desmond, 1995), the burden of marking on academics returns. 214 Australasian Journal of Educational Technology, 2004, 20(2) In contrast, Lehman and Herring (2003) describe the use of spreadsheets to provide formative feedback only (point 4 above). In their method, students enter formulas for rule based problems. Formative feedback is immediate since a correct formula is indicated by a change in font colour from red to green. The spreadsheet method described in this paper goes further in that it automates the summative marking with the resultant benefits to academic productivity as well as allowing immediate formative feedback like Lehman and Herring (2003). Moreover it reduces the propensity to cheat by providing each student with a unique file and data in their spreadsheet assignment. Figure 1: A student view of an individualised spreadsheet assignment 3. Context and assessment method The spreadsheet assignment marking method described here has been used in the University of Sydney’s Faculty of Economics & Business for over 10 years, first using Lotus 123, and then Microsoft Excel with macros written in Visual Basic for Applications (VBA) from 2001. Originating in accounting, the core discipline of the lead author who developed the Blayney and Freeman 215 concept and method, it has spread to other academics in the faculty and in professional accounting education. Accounting concepts are one area where students need to understand how different parts are calculated and how they interact. For example, the interaction of different assumptions for sales price and quantity sold when forecasting net present value needs to be carefully considered if one is making a business case to a financier. Figure 1 describes a typical spreadsheet assignment. The student is provided with an Excel file with raw data (here cells O16 to O22) required to solve a rule based problem (here cost-volume-profit). The student is required to enter cell referenced formulas to demonstrate their understanding of the rules in the concept (here cells O26 to O29). Instant help is available in two forms as indicated by the two buttons in Figure 1. A student can choose to see simply the final numerical answers to each required entry cell for their individualised spreadsheet assignment by selecting the button ‘Help me, I am stuck’. They can then work towards the correct rule based solution using cell referenced formulas. This is similar to having the answers in the back of a book with no workings. An alternative formative feedback is available with the button ‘Check my entries please’. Students selecting this option have typically made an attempt at demonstrating their knowledge of a concept by entering a cell referenced formula for the rule based problem. In choosing this button they obtain instant feedback on their progress, either textual or numerical (i.e. a mark) or both. Figure 2: A student view of feedback following self assessment 216 Australasian Journal of Educational Technology, 2004, 20(2) Figure 2 is a typical view of the individualised instant feedback provided to a student that completed the first two entries correctly. The feedback for entry 3 indicates that a correct value has been entered rather than a cell referenced formula. The feedback for entry 4 indicates that they entered the wrong formula and should try again. Overall feedback is also provided via a mark (here 3 out of 10) and a comment, in this case pointing the student to the textbook as their next action for improvement. Students can continue to attempt a problem until all entries are correct or time runs out and the file must be submitted. With minor additions, the same VBA code used to generate the formative feedback is subsequently used to mark all submitted files. There are two benefits of the assessment method. Automating the assignment marking process should improve student attitudes to learning since feedback is not only more immediate (Epstein, Epstein and Brosvic, 2001), but also improve learning since the feedback is more complete. Lehman and Herring (2003, p.332) note that the benefit over multiple choice tests is most obvious when answers are wrong - “for students unable to derive the correct answer, a single check figure does not provide an adequate level of feedback to guide students to the correct answer”. In this more sophisticated context, the extra steps taken to individualise the spreadsheets, by providing individualised data and different cell positions, means less incentives exist to cheat and improved attitudes to learning should also result in improved learning. There are of course other benefits to learning such as improved students’ personal computer skills and spreadsheet expertise. Figure 3: Development steps for self assessable individualised spreadsheets The second benefit relates to improved productivity for teaching staff. This method was developed chiefly as a response to the challenge of assessing Step 1 Question designed Step 2 Create Excel file with 3 worksheets Step 3 Add VBA forms, code modules Step 4 Create unique files for students Step 5 VBA code in Master Marker updated Step 6 Provide unique files to students Step 7 Students submit files for marking Step 8 Run Master Marker file. Post results. Blayney and Freeman 217 large numbers of student accounting assignments in an economical and timely manner, as well as an implicit recognition of the benefit to student learning of practice and immediate feedback. Automating the marking process has allowed assignments to be issued more frequently. Figure 3 shows the eight steps to generate individualised spreadsheets providing both formative self assessment and summative assessment. These are briefly elaborated below assuming Microsoft Excel as the worldware spreadsheet program. A forthcoming paper explains the specific detailed programming underlying the method. 1. Instructor designs assignment question. Instructors need to provide sample data, data constraints, identify the items requiring student answers and suggested solutions. Optimally this is provided in a spreadsheet file format with cells requiring answers to be highlighted. Suggested solutions for each required input cell using formulae with cell references must also be provided. This will allow the creation of VBA marking code that will ensure students learn the principles of good spreadsheeting, in particular the danger of embedded values in cells overriding global changes in data assumptions. 2. Instructor-programmer creates spreadsheet file with three worksheets. The master assignment template worksheet can be created from the information or file from step 1. Two additional worksheets created, but both are hidden. One displays self assessment results activated upon student request to assist the student learning, and another to record self assessment activity, used for evaluative research purposes. 3. Instructor-programmer adds macros in VBA code to the master assignment template worksheet to allow students to exercise the self assessment option. Each of the three VBA forms is standard to the assessment method and can therefore be imported into the master assignment template directly. Of the seven VBA code modules, four are standard to the assessment method and three require customisation for the individual assignment. These three modules provide the code that changes the question data to allow for testing if students have used proper spreadsheet techniques, namely the use of formulas rather than hard coded values since the former allow assumptions to be changed accurately. The VBA code simply compares the student’s entry cell outputs to the correct outputs with the changed data. Once these VBA modules have been added, the master assignment file should be provided to a discipline expert for rigorous checking. 4. Instructor-programmer creates a separate individualised file for each student. This is accomplished by taking a file containing a list of student names and numbers and invoking a VBA subroutine that performs a four step 218 Australasian Journal of Educational Technology, 2004, 20(2) procedure for each student on this list. It opens first the master assignment template file and pastes the current student’s name and student number to a visible location and the student number to a hidden, protected location. The student file is individualised further with the input and hiding of a random number (up to 10) of rows and columns. This randomisation of the location of an assignment’s data and entry cells makes it extremely difficult to use the cell copy function from another student’s assignment file. The assignment files can be further individualised by randomising the question data within given parameters. Finally the assignment template file is saved using a combination of the assignment number and the student number as the file name. At this point a variety of assignment files should be provided to a discipline expert for rigorous checking to ensure the data randomisation parameters used are valid. 5. Instructor-programmer creates formal summative marking master file. This is essentially the same code as step 3 above for providing informal self assessment feedback on student activation, but allows an additional outcome, namely individual student marks being collated in one file. Formal assessment of submitted assignments is performed by a marking master file containing a number of VBA code modules to perform the marking and a worksheet with a list of students and recorded marks. The marking master file has several standard modules used whenever the assessment method is used and thus requires no changes. It opens submitted files, records marks and moves assessed files to a different location. Whenever a new assignment file is used within the assessment method, two new VBA procedures are added. One relates to changing the question data while the second relates to the marking code for assessing the accuracy of the student’s entries for that new assignment. Both of these procedures contain essentially the same code as in the master assignment file that allows students to self assess their work. 6. Students access individual assignment files. These are provided on a self service basis through the faculty student file server where each student has their own subdirectory named with their student number. 7. Students complete and submit assignment files. Students can use an instruction pack to learn how to use the assessment method or assist each other via the online discussion forum. The self assessment option provides instant feedback on the student’s work for the entire assignment irrespective of the number of entry cells completed. Upon receiving a self assessment of 100% (or when the student decides that a lesser mark is good enough), students submit their individual file for formal assessment. Currently students at the University of Sydney use Blayney and Freeman 219 the Blackboard digital drop box for submission of their assignment work. An email system with assignment files sent as attachments to the university has also been effectively used. A requirement of the marking master file (whichever system is used) is that all assignment submissions are located in a single directory. 8. Instructor-programmer marks submitted files. The marking master file is usually run on a daily basis requiring less than five seconds to open, assess and record the mark for each submitted file. Students receive notice of their result by a file posted onto the course site in Blackboard – of course, they should already know their result by applying the self assessment option. 4. Method Research data was gathered from multiple stakeholders affected by the assignment marking method during the first half of 2003. The method was used in three subjects at the University of Sydney – 1,100 first year Financial Accounting students, 600 second year Management Accounting students, and 200 postgraduate Management Accounting students. Of the 10 weekly assignments over the semester, assignments 3, 5, 7, and 9 contributed to 5% of summative assessment and the others were solely for formative feedback or practice. Quantitative data on student learning expectations and experience was collected by surveys of the first year Financial Accounting students. While 900 students responded to a survey prior to their exposure to the novel assessment to gauge their expectations of it (82% response rate), only 230 students were surveyed after their experience due to administrative problems (77% response rate). Analysis of responses for significance between different demographic variables was conducted with chi-square tests. In addition, student usage data was collected through an automatic recording feature built into the assignment files. Each time a student used the self assessment option the day/time and assessment mark were recorded on a hidden sheet of the assignment file. Use of the assignment self assessment option was analysed for 8,600 assignment submissions and 89,000 self assessments requests over the three subjects. The number, timing and frequency of self assessments were analysed. Qualitative student feedback was gauged via a review of unsolicited comments from the online Blackboard discussion board. Data from other stakeholders was gathered from surveys with the eight participating academics plus interviews with relevant IT staff, teaching and learning specialists and the department head. 220 Australasian Journal of Educational Technology, 2004, 20(2) 5. Results We present the results of our research using a multi-stakeholder perspective, beginning with students, and then examining respectively the effects on staff, department and university. Effect on students In discussing the effects on students, first using quantitative information and second with the qualitative information, we obtain a reasonably coherent picture of their attitudes to the assessment method. Table 1 describes survey results prior to the first year students being exposed to the automated spreadsheet assessment method. It covers students’ demographics (section 1), prior experience with worldware (section 2) and perceptions of the value of feedback for their learning and expectations of using the automated spreadsheet assessment method. Table 1: Pre-exposure survey responses Section 1: Background demographics Hours/week paid work None 37% 1-5 hrs/wk 15% 6-10 hrs/wk 23% >10 hrs/wk 25% Home Internet facilities Phone modem 62% Cable modem 33% None 5% Main access to Internet Uni lab 13% Home 87% First language English 49% Other 51% Gender Male 53% Female 47% Overseas fee- paying Yes 29% No 71% Age <21 yrs 92% 21-25 yrs 7% >25 yrs 1% Disability status None 94% Visual 2% Hearing 1% Other 3% Social security benefits None 86% Austudy 7% Other 7% Section 2: Previous experience using worldware No experience A little experience Some competence Expert Spreadsheet 23% 45% 29% 3% Word processing 2% 11% 63% 24% Online threaded text discussion forums 43% 34% 18% 5% Emailing with attachments 7% 20% 43% 30% PowerPoint 23% 37% 32% 8% Blayney and Freeman 221 Section 3: Perceived value of feedback and expectations of assessment method (Strongly) Disagree (Strongly) Agree No opinion Feedback on my homework, quizzes & assignments is crucial to my learning 3% 94% 3% Working with others is a productive way to get feedback & learn 8% 90% 2% Model answers to past exams and quizzes help me gauge my ability 2% 94% 4% I make a serious attempt with past exams & practice quizzes prior to looking at answers 11% 82% 7% I expect the assignment with self assessment to be non-threatening for learning 10% 76% 14% I expect the spreadsheet assignments to be a productive way to learn 13% 76% 11% I have confidence that the computer will assess my work accurately 21% 64% 15% Sections 1 and 2 above reveal positive pre-conditions for introducing the automated spreadsheet assessment method. First, with 48% of students working 6 or more hours in paid employment, we would expect busy students to be more likely to want to complete assessment tasks and obtain feedback away from a classroom context. This is supported by the result that 95% of respondents have home computers and Internet access, although only 87% use their home equipment as their main facility. Second, we expected spreadsheet assignments to be an attractive option for 51% of students for whom English is a second language since there is less pressure to earn marks by demonstrating high order skills of argument and critical thinking in a second language. However, the low level of prior experience with spreadsheets (i.e. 68% reporting little of no spreadsheet experience) did reduce our expectations of the latter. Section 3 of Table 1 reveals this student group strongly supports feedback for learning (94%), whether working with others (90%) or by manually self assessing against model answers (94%), although not all make a serious attempt (11%) before viewing the feedback. However, despite the lack of prior experience with this type of assessment, the majority indicated they expected them to be non-threatening (76%), a productive use of their time (76%) and provide an accurate assessment result through the automated assessment marking method (64%). Chi-square analysis of the items in section 3 against different demographics revealed no significant difference based on gender (p=0.51). Similarly, and to our surprise, no significant difference was apparent for those working more hours in paid employment (p=0.47). Surprisingly, students speaking English as their first language regarded feedback as being more crucial to their learning than students with other native languages (p=0.01), and younger students valued feedback more highly than students over 21 years (p=0.05). 222 Australasian Journal of Educational Technology, 2004, 20(2) Table 2 details the survey responses of students after they have experienced the spreadsheet assessments providing automated formative feedback and receiving summative assessment results from their submitted spreadsheets being automatically marked by the method. Section 1 reveals that students had a reasonably clear idea of what to do (62%), that the workload matched the assessment value (64%), allowed sufficient time to understand and apply background concepts (81%) and that students tend to complete the weekly assignments in one or two intensive sessions (75%). A result of 71% completing it off campus is consistent with the pre-exposure survey where students reported a preference to do so. Section 2 shows that the administrative instructions provided to students to receive, complete and submit the spreadsheet assignment files were generally adequate (74%). However, a reasonable proportion of students indicated disagreement with the usefulness of the different forms of support provided, namely print (28%), online (32%) and lab supervisor (35%). Section 3 indicates the relative usefulness of different types of feedback available to students to complete the spreadsheet assignments. To some extent students perceived that they could do well without obtaining feedback since 71% indicated it was ‘a simple case of repeating work done in class’. Although encouraged to work in groups to give each other feedback to learn the concepts necessary to complete their assignments, few (26%) report doing so. The latter could reflect the high tendency to complete the assignment at home or possibly individualisation of the spreadsheet made it too complex to do so easily. While a greater reliance was placed on the feedback from the text (61%), students gave greatest credit (82%) to the automated formative feedback built into the spreadsheet self assessment options. Some 79% of respondents indicated that such feedback was a non-threatening way to assist their learning. Section 4 reports students overall perceptions of the automated formative and summative spreadsheet assessment method. The majority (76%) agreed that such assignments were a productive way to learn, helped them develop problem solving skills in the accounting subject area (71%) as well as their spreadsheeting skills (73%). The responses regarding the perceived accuracy of the marking program were somewhat disappointing, with only 55% agreeing that their work was assessed accurately, less than they expected prior to exposure to the method (66%). However, these results can be attributed to an error in the summative assessment marking code in one of the weekly assignment that resulted in students receiving a result of 98% while their formative assessment indicated they were 100% correct. Student dissatisfaction with this single error in the assessment method was reiterated in the free Blayney and Freeman 223 Table 2: Post-exposure survey responses Section 1: Assignment expectations and workload (Strongly)Disagree (Strongly) Agree No opinion I had a clear idea of what was expected for the assignments. 35% 62% 3% The workload required for the computer assignments matched the assessment value. 29% 64% 7% There was sufficient time to understand and apply background concepts 15% 81% 4% I completed each assignment in one or two intensive sessions 18% 75% 7% I completed the computer assignments away from campus 25% 71% 4% Section 2: Administrative instructions and support to do the assignments Overall (as a package) the resources provided for completing the assignments were adequate. 21% 74% 5% I found the instruction package and answers to FAQs useful. 28% 54% 18% The Blackboard Discussion board was useful. 32% 47% 21% The computer lab supervisor was a useful resource. 35% 31% 34% Section 3: Relative use of different types of formative feedback To do well on the computer assignment was a simple case of repeating work done in class. 23% 71% 6% I completed the assignments in a group to learn from others 66% 26% 8% I used the text and/or lecture notes while completing the assignments. 37% 61% 2% The self assessment options motivated me to keep trying 13% 82% 5% I found the spreadsheet assignment with the self assessment option to be non-threatening for learning. 13% 79% 8% Section 4: Overall perceptions I found these assignments to be a productive way to learn 20% 76% 4% These assignments helped develop accounting problem solving skills 22% 71% 7% The assignments helped develop my spreadsheet skills 20% 73% 7% The computer marking program assessed my work accurately 38% 55% 7% The computer assignments motivated me to learn. 33% 60% 7% These assignments were more enjoyable than normal ones 17% 75% 8% If possible all assignments should provide self assessment 10% 82% 8% 224 Australasian Journal of Educational Technology, 2004, 20(2) response section of a number of survey responses. This reinforces the need for absolute accuracy in the marking program as essential to maintain student confidence in any assessment method, but particular an automated one. This also probably tarnished the result for the motivational effect of the assignments (only 60%). However, despite this one off bad experience, students reported that the assignments were more enjoyable than ‘normal’ assignments (75%) and 82% want all assignments to have self assessment option. The analysis of the post-exposure survey responses by demographic variables reveals few significant differences. Three exceptions are: older students had a poorer opinion of the accuracy of the marking method (p=0.04); students that worked more hours in paid employment were less motivated to learn by the method (p=0.01) and had less confidence in the accuracy of the marking method (p=0.08). Table 3 summarises student free responses to survey questions of the three “best things” (n=294) and “suggested improvements” (n=170) for the automated spreadsheet self assessment and marking method. The overwhelming best aspect of the spreadsheet assignment is the ability to self assess (39% of free responses). Students expressed satisfaction with the ability that self assessment gave to gauging their learning and the ability to practise their learning. Novelty and flexibility were also favorably viewed. In terms of improvements, the greatest request related to increasing the overall assessment value for the assignments to justify the effort expended (32%). As previously noted, a minor programming error most likely explains the second most cited request for improved accuracy (18%). Better instructions (15%) was also highlighted in the survey questions as something to be addressed. Table 3: Student free responses of marking method Best aspects n=294 Improvements n=170 Self assessment enables ability to gauge learning and allows practice thus reinforcing key concepts learned 39% Greater assessment value for assignment justifying effort 32% Easy marks 16% Improve marking accuracy 18% Novel way to learn 12% Better instructions and format 15% Flexibility of IT assignment 8% Don’t require formulas 11% Development of spreadsheet skills 4% More detailed feedback 2% Other 21% Other 22% Analysis of the student messages within the online discussion board, an optional resource for students to obtain support from the instructor and peer students, provides additional support for the assessment method. Students used it to clarify understanding of how the spreadsheets worked Blayney and Freeman 225 and help them understand the concepts to complete the calculation questions. Most of the questions/comments related to administrative aspects, for example ‘Where are the files located on the network?’ and ‘How does the digital drop box work?’. There were several questions about how the spreadsheets worked, for example 'What kind of formula should I put in?', and a few about the underlying concepts being tested, such as 'Could anybody tell me how to calculate the average inventory as I can't find out the formula in the textbook?’ This suggests that students did not find the spreadsheet assessment requirements and method overly complex. Finally we turn to a discussion of how students used the self assessment facility. The popularity of the self assessment option is apparent from Figure 4. 0 25 50 75 100 1 11 21 31 41 51 61 71 81 91 Self assessment number % Mark received Figure 4: Use of self assessment Students used an average of 10.3 self assessments per assignment (namely 88,969 self assessments requested for 8,600 assignments submitted over three subjects). 47% used the self assessment option six or more times to get feedback on their progress during their completion of any individual assignment. This reinforces the high popularity of the formative feedback found in the pre and post-exposure survey. We believe the 15% that did not appear to use the self assessment option is due to students saving their file with a different name, say for backup purposes, during the completion process. Saving the file back to the required filename just before submission would lose any record of self assessments on the assignment since it is a ‘new’ file. Figure 5 depicts one extreme usage pattern where a student used the self assessment option 97 times in one of the weekly assignments. It also indicates two dips where the student received feedback that their progress in producing a complete solution was negative and unless corrective action was taken would reduce their mark. 226 Australasian Journal of Educational Technology, 2004, 20(2) 522 424 269 212 162 85 58 835 1295 3267 1258 0 1000 2000 3000 0 1-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 Number of times self assessment used for an individual assignment file Number of files Figure 5: An extreme self assessment usage pattern Effects on academic staff The lead author who developed the method, has retained and enhanced the automated assessment method because it has provided him with productivity benefits and a strong sense of satisfaction in helping students learn through practice, and helping his colleagues to realise productivity benefits. The main difficulties encountered by the developer are an exposure to the vagaries of IT hardware systems and software upgrades (e.g. time wasted when students submit spreadsheet files from earlier versions of Excel). Additional problems arise when academic colleagues using the method provide inaccurate or insufficient description of concepts; program relationships between concepts inaccurately, give insufficient time to quality assurance, or make incorrect assumptions about core knowledge with which they are unfamiliar. Views by participating academics were gauged from ongoing informal interactions as well as a short survey. Table 4 indicates their perceptions of the best aspects and those most in need of improvement that were gleaned from the free response comments. Table 4: Academic perceptions of automated assignment marking method Best aspects n=8 Most needed improvements n=8 Timely feedback to students 80% File submission 80% Allows regular assignments without costly marking 60% Marking accuracy 60% Developer’s familiarity with content facilitated integration 40% More feedback as to why answers are wrong 40% Frequently mentioned comments were the benefits of feedback to student learning and the productivity benefit to themselves, particularly in those cases where the lead author and developer assisting them was familiar with their content area. Improvement in the file submission process was requested most frequently. Blayney and Freeman 227 Participating academics were also surveyed on their perceptions of Rogers (1995) five innovation diffusion factors that affect the potential for further mainstreaming. Table 5 indicates that all participating academics perceived that the method had relative advantages over other assessment options. While the majority perceived that the innovation was compatible with institutional context and culture, its introduction would be unlikely to happen without the assistance of the developer who specialised in the process and had VBA expertise. There was also scepticism about the ability to trial the method without investing in full adoption and scepticism about the visibility of benefits to non-participating colleagues. Table 5: Perceptions of potential for further mainstreaming Rogers’ five factors affecting innovation diffusion Agree Dis-agree Unde- cided 1. Overall, I can see several relative advantages of the automated assignment marking method over other options for assessment and feedback in terms of time, cost, effectiveness, quality of results (relative advantage) 100% 2. The automated assignment marking method is acceptable and compatible with the existing academic culture, practices, values and needs (compatibility) 75% 25% 3. It was easy for me to try the automated assignment marking method without actually making a decision to adopt it into my course (trialibility) 37% 63% 4. While the automated assignment marking method is easy to understand holistically, it would be too complex to learn and use effectively without assistance (complexity) 87% 13% 5. The potential benefits of the automated assignment marking method are visible for my academic colleagues considering adopting it (visibility) 36% 64% Effects on department/school The effects on the department can be gauged from feedback provided informally and formally to the developer. The most common benefit attributed to the method is the improved productivity for academics, in conjunction with the improved learning outcomes. The Head of School focussed strongly on the productivity aspect noting the “benefit of this software is highlighted in an environment where the university is seeking to balance research outcomes with teaching excellence. Clearly the method provides an efficient marking mechanism while freeing up academic’s time to pursue better research outcomes and a better understanding of current teaching directions”. A faculty teaching and learning specialist commented very favourably about the benefits to students, especially the formative feedback aspect, and her own improved productivity arising 228 Australasian Journal of Educational Technology, 2004, 20(2) from the assistance of the developer. Other benefits include the improved intra-departmental cooperation, the enhanced reputation of the faculty and disincentives for students to cheat. IT staff, including those who interact with students on the helpdesk, are also positive. While the described method has been used in a range of business disciplines (e.g. financial accounting, management accounting and tax) it could easily lend itself to use in other higher education disciplines such as maths, statistics, engineering and physics. Attributes of the assignment marking method that require more careful monitoring are systems failure, particularly because the method relates to assessment, and the risks due to skills tied up in a single academic specialising in the method. Effects on university The marking method has the potential to provide the same benefits to the university that it did for the department and faculty where it has been used for the last ten years. Two potential risks to the university are cited in discussions with university administrative and academic managers: reliance on one specialist to provide the service, and systems failure. The university’s exposure to one specialist can be overcome since the underlying software is a worldware spreadsheet program that is widely used and understood. A minor negative aspect of the marking method relates to file server storage requirements. Maximum usage was almost 10 gigabytes used by the marking method in the next to last week of the semester. This usage was approximately evenly split between the server location providing the individualised assignment files to students and a “submitted and marked” folder containing all student assignment submissions. For the close to 2,000 students using the marking method, server requirements average about 5 megabytes per student. Should storage pose a problem, the marking method’s disk storage requirements could be reduced by removing individualised files from the “self service” location after students have downloaded their assignments, or storing marked files for a limited time. Currently all submitted files have been stored indefinitely for research purposes and to respond to rare student queries. 6. Conclusion This research was motivated by a call from Higgins et al (2002) that feedback is under-researched. We report the effects of providing students with the ability to self assess their individualised spreadsheet assignments, at their own discretion and on a weekly basis, and the effects on academics Blayney and Freeman 229 from automating the summative assessment process using virtually the same programming code. We expected that this would improve students’ attitudes to learning and provide opportunities to learn, without increasing the load on academics. Strong support for the notion that timely and effective feedback is a crucial element of the learning process has been obtained from a variety of sources; the student surveys pre- and post-exposure to the method, comments from the online discussion board and the widespread use of the self assessment option. However, we caution against generalising the result that the automatic feedback via spreadsheet assignments is ‘more enjoyable than normal’ assignments (75%) given the documented Hawthorne effect explaining higher satisfaction and performance simply due to being the focus of a study (Handy, 1999). There was also strong support for the productivity benefits to participating academics. Barriers to mainstream diffusion of this innovation within the university context where it has been developed relate primarily to three of Rogers’ (1995) identified factors, namely complexity, trialability and visibility. In terms of complexity, the necessary specialist skills in VBA for Excel that are required to obtain the full benefit of the automatic marking and self assessment options are too onerous for most individual academics to acquire. However, as the software is worldware, universities are likely to have access to the necessary skills. Reusable objects may be part of the future solution. Finally, academics perceived that the benefits were not entirely visible, although they were unanimous in recognising the relative advantages. This research contributes to the process of making the spreadsheet method and its benefits for learning and productivity more visible. Future research with the automated spreadsheet assessment method is concerned with identifying effects of different types of self assessment options, such as limiting the number of times students can access self assessment. Other areas worthy of research include effects of assignment difficulty on self assessment option, interactions between assignment value, assignment frequency and cheating with self assessment usage, as well as exploring the generalisability to other contexts. References Alexander, S. & McKenzie, J. (1998). An evaluation of information technology projects for university learning. Canberra: AGPS. Biggs, J. (1999). Teaching for Quality Learning at University. Buckingham: Open University Press. 230 Australasian Journal of Educational Technology, 2004, 20(2) Elliot, C. (2003). Using a personal response system in economics teaching. International Review of Economics Education, Economics Learning and Teaching Support Network. [verified 30 May 2004] http://www.economics.ltsn.ac.uk/iree/i1/elliott.htm Chickering, A. W. & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. American Association for Higher Education Bulletin. October pp. 3-6 [viewed 31 Dec 2003 at http://www.aahe.org/technology/ehrmann.htm, verified 30 May 2004 at http://www.tltgroup.org/programs/seven.html] Edmonds, T., P. & Tsay, B. (1989). Using Lotus 1–2–3 macros as grading assistants. Journal of Accounting Education, 7(2), 271-278. Epstein, M.L., Epstein, B.B. & Brosvic, G.M. (2001). Immediate feedback during academic testing. Psychological Reports, 88, 889-894. Handy, C. (1999). Understanding Organisations. Penguin. Higgins, R., Hartley, P. & Skelton, A. (2002). The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27, 53-64. Izard, C.D. & Reeve, J.M. (1986). Electronic spreadsheet technology in the teaching of accounting and taxation - Uses, limitations, and examples, Journal of Accounting Education, 4(1), 161-175. Lehman, M.W. & Herring, C.E. (2003). Creating interactive spreadsheets to provide immediate feedback, Journal of Accounting Education, 21(4), 327-337. Prosser, M. & Trigwell K. (1999). Understanding Learning and Teaching: The Experience of Higher Education. Buckingham: Open University Press. Ramsden, P. (2003). Learning to Teach in Higher Education. 2nd ed. London: Routledge. Rogers, E. (1995). Diffusion of Innovations. 4th ed. Free Press, New York, 1995. Russell, T.L. (1999). The 'No Significant Difference' Phenomenon. [viewed 31 Dec 2003 at http://teleeducation.nb.ca/nosignificantdifference/, new URL verified 30 May 2004 http://www.nosignificantdifference.org/] Smith, E. & Coombe, K. (2000). Distance education at arm's length: Outsourcing of distance education marking. Competition, Collaboration, Continuity, Change in Distance Education Conference. Adelaide [viewed 31 Dec 2003, offline 30 May 2004] http://www.com.unisa.edu.au/cccc/papers/refereed/paper46/Paper46-1.htm Teets, W.R. & Desmond, N. (1995). Anti-cheating 201: Preparing multiple versions of accounting exam problems through dynamic linking of word processing and spreadsheet programs. Journal of Accounting Education, 13(3), 343-348. Blayney and Freeman 231 Vantage (2003). [verified 30 May 2004] http://www.intellimetric.com/demosite/demo.scriptedessay.html Paul Blayney Faculty of Economics and Business The University of Sydney, NSW 2006, Australia Email: p.blayney@econ.usyd.edu.au Associate Professor Mark Freeman Director, Centre to Advance Learning in Economics and Business Faculty of Economics and Business The University of Sydney, NSW 2006, Australia Email: m.freeman@econ.usyd.edu.au http://www.econ.usyd.edu.au/caleb/