Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 121 Use of New Methodologies for Students Assessment in Large Groups in Engineering Education B. Tormos, H. Climent *, P. Olmeda, F. Arnau CMT-Motores Térmicos, Universitat Politècnica de València, Camino de Vera s/n, 46022, Valencia, España, Tel. 963 877 650, Fax 963 877 659, hcliment@mot.upv.es* Received: 2013-12-15; Accepted: 2014-01-28 Abstract In this paper, a student evaluation methodology which applies the concept of continuous assessment proposed by Bologna is presented for new degrees in higher education. An important part of the student's final grade is based on the performance of several individual works throughout the semester. The paper shows the correction system used which is based on using a spreadsheet with macros and a template in which the student provides the solution of each task. The employ of this correction system together with the available e-learning platform allows the teachers to perform automatic tasks evaluations compatible with courses with large number of students. The paper also raises the different solutions adopted to avoid plagiarism and to try that the final grade reflects, as closely as possible, the knowledge acquired by the students. Keywords Assessment methodology, large groups, technical subjects, continuous assessment. mailto:hcliment@mot.upv.es Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 122 1. Introduction During the development of new degrees in higher education in Spain, new subjects have to be adapted to Bolonia’s philosophy and have to include continuous assessment systems (Shipman et al. 2003; Morales et al. 2012; Christoforou and Yigit 2008; Dixon et al. 1987). These systems are easily applied in subjects of latest academic years, where the number of students in the subject is reduced and the required effort in the evaluation process is reasonable (Heywood 2000; Shorter and Young 2011). However, in the case of subjects in first or second courses, a large number of students (115 approx.) is usually found in Spanish universities. The adoption of a continuous assessment in this scenario is highly time consuming for the teachers, who have to reduce their dedication to research and management activities. The present paper describes an experience in the use of new methods for the evaluation of different tasks developed by students in a subject with a large number of course assistants. 2. Background The methodology presented in this paper is applied in the subject “Thermal Engineering”, which starts in the second semester of the secod course in the Mechanical Engineering Degree in the Escuela Técnica Superior de Ingeniería del Diseño (ETSID) from the Universitat Politècnica de València (UPV). 2.1. Subject description It is important to remark some characteristics of the subject since the developed methodology may not be effective for subjects that do not fulfill some requirements. The number of students is approximately 230, split in two groups of similar size. The Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 123 subject, with 4.5 ECTS credits, combines theoretical lectures, problems solving in classroom and laboratory activities. The students’ assessment system consists of several evaluation acts which are detailed in Table 1 and fulfill the requirements given by the University guidelines. The written exam is a limited-time activity that takes place at the end of the course. It consists of numerical problems and the students do not have access to books or additional material during the performance of the exam. There is another opportunity two weeks later, with the same characteristics, for students that did not pass the ordinary exam. There are three tests for the evaluation of each part of the subject. These tests are fulfilled using the on-line e-learning platform developed by the UPV, which will be described in the next section. The tests are multiple-choice tests with only one correct answer and can be reached in the e-learning system during three consecutive days. Once the students open the test, they have limited time (usually half an hour) and there is no possibility to save and continue the test in other moment. Half of the points in the qualification mark can be achieved with individual tasks, where the students present numerical exercises similar to those solved in class. These tasks are evenly distributed all along the semester and each one has its own deadline in order to promote a continuous learning process in the students. Each task involves a duration of 3 or 4 hours and should be performed individually, although the students may have access to additional material if they want. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 124 Table 1. Students assessment system in the subject “Thermal Engineering” Evaluation method Evaluation act Weight [%] Written exam: 30% Ordinary Extraordinary 30% (30%) Tests: 20% Test 1 5% Test 2 10% Test 3 5% Individual tasks: 50% Task 1 15% Task 2 10% Task 3 10% Task 4 10% Task 5 5% By observing the weights of the different assessment methods in Table 1 it is observed that only the 30% of the qualification mark is achieved under teacher controlled conditions. The other 70% relies on the students’ code of honor. In order to avoid plagiarism and provide qualification marks that correlate with the students’ acquired competences, an additional written activity is included in the evaluation system; it is called the “Task checking test”. This additional test is carried out in a similar way as the written exam. However, the contents of this test are very different. Specifically, this test is designed in such a way that it is quick to perform and to correct. It consists of a collection of identical paragraphs to the students individual tasks. The aim is that the results of this test provide very high values for students who have actually carried out their tasks with interest and rigor, and mediocre or very low values for the rest of the students. The Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 125 result of this test, on a scale between 0 and 1, directly multiply the grade obtained in the individual tasks, so that it is impossible to pass the subject by being evaluated only with methods were the authoring cannot be perfectly verified. 2.2. E-learning platform in the UPV PoliformaT is the e-learning platform that gives support the teaching activities in all the subjects of the UPV, which is a member of the educational project Sakai. Using this platform, students and teachers relationship breaks the traditional barriers. PoliformaT is the UPV adaptation from the Open Source Sakai platform, used worldwide by more than 100 universities. PoliformaT has a lot of features but, concerning the purpose of this work, only two of them are important to describe: tasks and grades sections. The tasks tool allows the creation, distribution and correction of the students’ activities. Students’ activities are private and can only be seen by the student and the teacher. On the other side, grades tool allows the calculation, saving and distribution of the students marks. Information related to students’ marks in the tasks tool is transferred directly into the grades section, where the students can follow the scores as they get their tasks and tests corrected. 3. Methodology for students’ tasks assessment As described before, each one of the five tasks consists of similar numerical exercises to those solved in class throughout the teaching units that make up the subject. A classic correction (such as that carried out when correcting a written test) of the five tasks is not reasonable to perform due to the large number of students involved. Thus, the tasks are evaluated taking into account only the final numerical results when solving each Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 126 problem. It is advisable to divide the exercises that make up a task on a large number of independent sections so that the student can easily detect where in the resolution process has failed. A task is created when a lesson is finished and the task typically remains open for about three weeks. In this way, the students can organize themselves and manage the other subjects of the course. The creation of the task is carried out in PoliformaT and includes two files: a text file containing the instructions and the exercises and a spreadsheet file that serves as a template for the presentation of the results. 3.1. On the instructions of the task and deadlines The text file is twofold: to clearly state the task instructions (with indications on how to deliver it), and on the other hand, to present the set of exercises to solve. In relation to the latter, and in order to try to reduce the cases of plagiarism, it is remarked that the input data of the exercises are modified with each student (they depend on the personal passport number). There are two important dates concerning the delivery of the task. On the one hand, a task delivered after the “deadline date” will not be corrected. This deadline is established at the time of creating the task in PoliformaT platform, which will prevent a student to upload the task. In addition, the “recovery date” is defined as a date prior to the deadline (around a week before). All the delivered tasks prior to the “recovery date” will be corrected and the results of its evaluation transmitted to students. Students have now the opportunity to amend the results where they have made a mistake and re- upload the corrected task before the deadline. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 127 3.2. On the spreadsheet template The spreadsheet helps the students to enter their numerical results in the appropriate cells. In this way, the correction process can be carried out as automatically as possible. To do this, it is advisable to give this spreadsheet the following: (1) it must be locked in order to avoid inserting and deleting rows and columns, except in cells where the end result of each exercise section is expected, (2) it is advisable to have a cell showing the input data calculated from the passport number (and should not show anything if the student has not entered the value of the passport number in the corresponding cell), and (3) it is advisable to include errors handling in the cells fulfilled by the students (typical examples are: using the wrong decimal symbol or confusion between the capital “o” and the zero digit). Once the students have fulfilled the template with their numerical results, they upload the spreadsheet file to PoliformaT platform and wait for the correction, either on the “recovery date”, either at the “deadline date”. It should be noted that a student can upload a file to the application, even if it is not completely solved. However, since the server is able to store several files, it is important to say to the students in the delivery instructions that there must be only one file before the correction date. Otherwise, the system is unable to know which is the last file that was uploaded, as that information is unfortunately lost. The use of a reasonable filename to the uploaded file to the server is also a good practice. 3.3. On the correction process Once the “recovery date” or “deadline date” have passed, the correction process starts. PoliformaT greatly eases the management of the tasks delivered by the students. The process is described in the following paragraphs. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 128 Firstly, the teacher enters the tasks section in PoliformaT and downloads all the files submitted by students. PoliformaT allows a massive download in a compressed ZIP file with all the information. This action, along with the extraction of the compressed file in a folder on the local hard drive, usually takes about five minutes. The extracted information is structured as follows: (1) a main folder named the same way as the task, (2) a Microsoft Excel© file, called “grades.xls” which contains a list of all students, and (3) a folder for each student called after his name, first name and passport number in brackets. Within each student folder there is a subfolder called “Task Attachments” which contains the spreadsheet template file with the numeric results. Then, every student file is corrected in an automatic procedure using a spreadsheet with internal macros programmed in Visual Basic©. The basic functions that are programmed for this automatic correction are able to: (1) create a file for each student in a subfolder within each student folder called “Task Comments”, (2) compare the student's solution and indicate in the file if the exercise’s section is correct or not, (3) manage potential standard errors (most of them are related to problems in the units and their detection can be carried out automatically), (4) calculate the final mark based on the errors in the different sections, and (5) insert the mark in the corresponding row in the global file “grades.xls”. This process usually takes about ten minutes. Finally, once the correction process is finished, the teacher must make a compressed ZIP file containing the main task folder and upload the resulting file to the task section in PoliformaT where the students’ marks are published. The time spent on this process is around five minutes. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 129 4. Results and discussion Despite the large available period for tasks delivery, students generally tend to deliver the tasks in the last moment. In Figure 1 the number of deliveries for two different tasks are shown as a function of time. They are split depending on “recovery date” or “deadline date”. Task 1, shown above, was open for 30 days while task 2, plotted below, was open for 19 days. It can be clearly detected when the “recovery date” was scheduled in both tasks (day 12 in task 1 and day 14 in task 2). It can be observed a second peak when the task deadline occurs. Figure 1. Daily deliveries in two different tasks. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 130 Figure 2 shows the histograms of the marks obtained in the previously described tasks 1 and 2. We present both the mark achieved before “recovery date” and before “deadline date”. A significant improvement in the students marks is detected thanks to the feedback provided in the first correction process. In the first task, it is observed that only 21 students from 152 deliveries before the “recovery date” achieve the maximum qualification. In the second task, the results are even lower; only 4 students (from 114) reach the maximum mark. The rest of the students have the possibility of making another delivery including the appropriate corrections before the “deadline date”. Figure 2. Marks histograms for tasks 1 and 2 depending on recovery and deadline dates. Figure 3 shows a histogram of the mark improvement between tasks presented before “recovery date” and “deadline date”. High values are found in the 2 and 3 points. It is also remarkable the high percentage of students that do not improve their marks (observed in the values given in 0 points), either because they are not able to solve the task properly, either because they settle for their first mark. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 131 Figure 3. Marks improvement between recovery and deadline dates. As already mentioned, activities outside the classroom control can give rise to plagiarism. This is an undesired effect that should be minimized for educational and impartiality reasons. The “Task Checking Test” aims to correct this possible mismatch between acquired knowledge and attained mark. The left plot in Figure 4 shows the results of the “Task Checking Test” and the average mark of the individual tasks. A very low correlation exists between the two variables. The right plot presents the variation in the obtained mark between the “Task Checking Test” and the tasks average. Positive values mean that the student has obtained a better mark in the “Task Checking Test” than in the tasks average while negative values means the opposite. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 132 Figure 4. Correlation between the “Task Checking Test” and the average marks of the individual tasks (left) and histogram of both marks difference (right). Interesting results can be obtained from the plot on the right. Half of the students present a marks difference between -1 and 1 point. 10% of the students have a better mark in the “Task Checking Test” than in the average of the individual tasks. 20% of the students have -2 or -3 points difference and the other 20% have larger differences between both marks. 5. Conclusions We have developed an automatic correction process for numerical exercises, whose input data vary with each student, based on the use of a programming environment within a spreadsheet file. The application of this system is advisable in subjects with large numbers of students and where there are several tasks to correct. The time spent by the teacher when correcting a task with 200 students is around 40 minutes (including two corrections: recovery and deadline dates). Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 133 Concerning the application of this methodology to a technical subject in second course following conclusions are obtained: Tasks deliveries occur on the same day as the deadline date. For this reason it is very convenient to include the “recovery date” with teacher’s feedback, which promotes the continuous learning process of the subject. Around 75% of the students make use of the correction before the “recovery date”. A very high percentage of these students make a second delivery before the “deadline date”. The mark usually increases with the second delivery. Although the values depend largely on the performed task, it is possible to state that the mark rises up to 2 or 3 points (in a scale from 0 to 10) in the revised student delivery. Lastly, since there is a low correlation between the final mark and the average marks of the individual tasks, it is advisable to include the “Task Checking Test”, which balances the weight of the marks obtained in the individual tasks. It is a quick exam (not only for the student but also in the correction process) that tries to capture plagiarism cases in the evaluation acts in non-controlled situations. In our experience, up to 20% of the students fall into this plagiarism group due to the high discrepancies between the mark in the “Task Checking Test” and the average mark in the individual tasks. Moreover, if this checking test would not exist, around 20% of the students would have 2 or 3 points more in the average mark of the tasks than what they deserve. References A.P. Christoforou and A.S. Yigit (2008). Improving Teaching and Learning in Engineering Education through a Continuous Assessment Process, European Journal of Engineering Education, 33(1), 105. DOI: 10.1080/03043790701746405. R. Dixon, G. Rawlings, E.A. Jones (1987). Experiences with Continuous Assessment, Assessment and Evaluation in Higher Education, 12(1), 24. Multidisciplinary Journal for Education, http://dx.doi.org/10.4995/muse.2014.2198 Social and Technological Sciences Tormos et al. (2014) http://polipapers.upv.es/index.php/MUSE/ Mult. J. Edu. Soc & Tec. Sci. Vol. 1 Nº 1 (2014): 121-134 | 134 J. Heywood, Assessment in Higher Education, Jessica Kingsley Publishers Ltd, Londres, (2000). D. Morales, G. Centeno, C. Vallellano, F. J. Doblas, A. J. Martínez-Donaire, A. Estévez, F. J. García-Lomas, Sistema de autoevaluación continua en la asignatura Tecnología de Fabricación, XX Congreso Universitario de Innovación Educativa en las Enseñanzas Técnicas, Universidad de Las Palmas de Gran Canaria, (2012). D. Shipman, S.L. Aloi, E.A. Jones (2003). Addressing Key Challenges of Higher Education Assessment, Journal of General Education, 52(4), 335 N.A. Shorter, C.Y. Young (2011). Comparing Assessment Methods as Predictors of Student Learning in an Undergraduate Mathematics Course, International Journal of Mathematical Education in Science and Technology, 42(8), 1061.