http://www.sajim.co.za/peer4.asp?print=1 Peer Reviewed Article Vol.2(2/3) September 2000 Assessment over the Internet Neill G. Harris Dept of Electrical Engineering, University of the Witwatersrand, South Africa n.harris@ee.wits.ac.za Contents 1. Introduction 2. Problems complex and confounding 3. Problems and solutions 4. Definition of response 5. Problem understanding and responses 6. Subject knowledge and concepts 7. Sentences and concepts 8. Instructor and computer co-operation 9. Student and assessment 10. Assessment assessed 11. Conclusions 12. References 1. Introduction In many circles, teaching and learning processes are being aggressively analysed and hotly debated. Much of the debate seems to centre around fairly assessing students' performance, whereas the educator, particularly at universities of academic standing, is really not concerned about performance but understanding. Unfortunately, performance is only an indirect and not necessarily reliable representation of understanding. In mathematics in the USA, as in many countries, faculty are complaining about the level of understanding of mathematics of students from high school (AMS 1996a, 1996b, 1997). This concern is detectible under the surface in many disciplines. Another major difficulty lies in translating the aims of a course as they appear elegantly phrased on paper to a fair and true reflection of the understanding of students who pass or of those who pass well. 2. Problems complex and confounding This disquiet and other activity in education have lead us to a great expectation of and reliance on assessing the outcome of a course, that is what the student should be capable of doing successfully on conclusion of the course. We assess the performance of the student against the outcome criteria and many people are satisfied at that point. The performance, top correctly assessed is a very valid criterion for examination. However, as many will immediately point out, in many cases correctly and disparagingly, it does not necessarily indicate understanding. What it does indicate is how well the student was able to represent the solution to the problem, be it from rote learning, pattern recognition, algorithm or understanding. In this article assessment focuses on two aspects: The assessment of understanding of the problem; and the assessment of understanding of the solution. The problem of assessing understanding is complex and self-confounding because of indirect measurement and difficulties arising from students expressing solutions in sentences. This means that the measurement needs to: accurately reflect the meaning of the student’s expressed solution; filter the results of assessment, such as the solution of a problem, to establish the component related to understanding; and ensure that the mechanism fairly tests the understanding being measured. Another hotly debated issue relates to allocating marks. Does one allocate part marks when a logical link in a solution is missing or incorrect? Philosophically, and in reality, the solution is incorrect from the first incorrect logical link onwards, and so no additional marks should be allocated. The contention is that this is the point at which understanding breaks down and subsequent evidence in the student’s solution does not demonstrate understanding, but the lack thereof. However, in an attempt to be fair, markers assign part marks because it appears that the student does understand more about the problem than the evidence indicates. This implies that we need to refine the mechanisms of our assessment of understanding to assess the student’s understanding of the: problem; and separately solution. In a following section, the method we have been trying to use to establish understanding of the problem is discussed. This however, is merely an extension of the normal method applied to assessing the solution. The next step is to develop a model that would allow a computer to mark a sentence of a solution. This is presented following a discussion of the terms used. Written material conveys information in its content and structure. Hence, there is a need for specific definitions of question, answer and a model representing a sentence that a computer can read and understand. 3. Problems and solutions Since terms, such as 'exam question', may refer to different concepts, in this case, an actual question or a problem, it is important to define the terms 'question', 'answer'¸ 'problem' and 'solution'. While they are loosely used in the literature, it is important to differentiate between them to establish the structure intended in each. As used in this paper, a question asks for a fact as an answer. An answer is defined as a single fact. A list of facts is defined to be the same as a single fact. A fact cannot be partly known. It is either known or unknown to the student. Several consequences arise from this top definition: There can be no part marks for an incorrect answer under this definition. Questions and problems, as defined subsequently, should not be mixed in an exam section. Answers need carry no more than one mark per answer. There is only one correct answer, provided the question is correctly and unambiguously phrased. It is debatable whether some facts are more significant than others, which is what is implied when the marks for different answers are weighted. There is no structure in an answer. This means that an answer is either a correct or incorrect fact. An answer may be a concept, but its premises are not implied in the question. On the other hand, the existence of a set-problem implies that a complete solution has been developed. A problem has a result that is part of the problem statement and the solution. A correct result is not necessarily part of the solution. The solution is a collection of correctly associated concepts that lead to a result. Thus, a solution can be partly correct, whereas an answer cannot be. In addition, a solution has: structure; assumptions; and a single result, but possibly more than one solution. Should a problem have several intermediate results, then it is a series of dependent problems. Further, a problem implies a solution. In fact, to be valid the solution must be compiled before the problem can be validly expressed. Thus, if a problem expresses the assumptions and result, as in a specification, then it is a solution with the linking structure removed. 4. Definition of response For credit to be awarded for understanding in a solution, the problem must be understood, as mentioned previously. Examiners sometimes set a problem that is similar to a problem already encountered by the students, but with a twist in it with its solution. This certainly indicates whether the student understands the problem and solution to some extent, particularly, as frequently happens, when students do not realize there is a twist. To address the difficulty of students not understanding the problem, my colleague, Dr Wigdorowitz, and I have been using two approaches. We require a student to make a response to the problem statement. We use the term response so that it is not confused with the solution to the problem. In fact, in the first approach, the solution is given! 5. Problem understanding and responses In the first approach, the problem statement includes an incorrect solution. The problem statement tells the student that it is incorrect. The student has to respond by understanding the problem and determining why the solution is incorrect. We have found this approach very effective because the correct response need only identify where the solution is incorrect. top top This immediately sorts out those students who understand the problem and those who do not. In addition, the responses tend to be concise and terse. Long-winded paragraphs are seldom received because the student is not able to apply what he does know correctly, but use his understanding to find out why it cannot be applied. Philosophically, this type of problem may be considered unfair in some ways. For example, a problem solution may be incorrect because the solution goes out of the bounds of the concepts encountered in the course. However, we have found that this is precisely why it proves, we believe, to be such a good measure of understanding. Our experience indicates that problems with solutions based inside the boundary of application of known concepts merely exercises the application of the concept. It is only as one attempts to probe the boundary of the concept and goes beyond it that one really begins to understand the concept properly. The second approach requires the student to express his/her understanding of the problem. We have found this type of response very difficult to assess because many students respond by rephrasing the problem using the same key words given in the problem statement. This, of course, tells us more about their ability in English than their understanding of the problem. Moving on to using the power of the computer to assess responses, the information content of a sentence needs to be modelled. 6. Subject knowledge and concepts Knowledge of a topic is expressed in concepts by means of a language. So, it is necessary to determine the relationship between concepts in the topic and then develop a means to relate the structure to that expressed in a sentence in language or a mathematical equation. To build a conceptual model for computer assessment, the precise meaning of terms, such as sentence, concept and structure, needs to be defined. Once a model is developed, programs can implement the model and be used in assessment. A concept represents an object that can be identified by a label, such as a name or symbol. An assumption is a concept that must be known, but not necessarily understood, and available to establish a concept. Availability, in this context, means that a concept must be known and the relationship to other concepts also known. For example, if a concept is known in mathematics from topology, but there is no means of relating it to concepts of structure, then that concept is known but unavailable in dealing with concepts of structure. A premise is a prior understood concept. A link is a concept that links two concepts in one direction. In a link, the associative property is important, whereas the concept or function property is relatively unimportant. Concept, premises and links have the following properties: A concept may be built up from two or more premises and two or more links. If there is only one linked premise, then the concept is effectively the same as the premise. A premise is simpler than a concept by definition. A link is a concept whose association property is significant. A concept has two or more incoming links from two or more premises. An assumption is a premise that has no incoming link. A concept has no outgoing link. top top 7. Sentences and concepts Since knowledge of a subject is most frequently expressed in language, it makes sense to model language expression in such a way that the computer can make comparisons of a model solution with those submitted by students. 8. Instructor and computer co-operation Our research shows that the computer and the instructor in a course must both be made to work harder and more closely together to achieve some success. Essentially, the computer has to identify the structure in the language of a solution so that the computer can match and mark it. An instructor compiles problems and solutions using a special editor. The editor opens a problem development screen (PDS) that is divided into three sections: problem initial conditions, solution section and the result section. The instructor may add sentences into each section in any order at any stage. Based on word processor operation, the editor identifies concepts and prompts the instructor to identify the structure. The structure is then automatically mapped into a database. When the solution is completed, the assessment application displays a graph of the concepts in the solution. The instructor then looks for duplication or breaks in the structure and attempts to correct them in the PDS. The graph has several benefits: It shows where links and premises are missing; and the problem statement can be checked to ensure that the initial conditions are sufficient. The graph can indicate the complexity of the solution by the degree of serialism, parallelism and meshing. Once the solution is consistent, a thesaurus is used to expand the problem database. A mark definition screen (MDS) quickly guides the instructor through the solution enabling him/her to allocate marks to each section or step. 9. Student and assessment The solution editor (SE), problem and mark definitions and database are down-loaded to the JAVA-enabled browser in the computer of the student. The student enters his/her solution using an editor, which is a very small subset of the editor used by the instructor. The marking applet marks the solution as it is developed. If it cannot resolve a concept or link, it flags it as incorrect, assigns a provisional mark, and the solution is uploaded into the assessment server when next the student logs on. The instructor is asked to evaluate the unresolved flags using the editor. This may either result in the problem database being expanded or the solution marked incorrect. The result is sent to the student. top top Marks may be allocated for the correct solution-so-far, or by part-marks. The student can call up a diagnostic screen that provides details of where errors were made. However, it intentionally does not provide the corrections or solution to the student. 10. Assessment assessed Assessment of written material still proves to be extremely effective. But, as many have pointed out, there are several limitations that must be recognized in arriving at fair assessment. 'Fair assessment' means that it accurately reflects the learner’s understanding of the content assessed or examined. This definition incorporates some key issues: The assessment is an objective expression of the learner’s understanding. The work represents the learner’s own understanding. The problems or task employed is fair in terms of content and complexity. The marking is fair in that it gives credit where it is due. Pressure and concern for fairness towards the student are both hampering sound assessment and improving it. For example, solutions are clearly set out and marks allocated for specific items of information and understanding. One sets out to establish the level of understanding of the student of the implication of a fact and ends up giving marks for the fact itself, because the course did not include the implication and the students did not know how to draw out or even express the implication. Our method of assessment by computer requires very close cooperation between the instructor and the computer in compiling problems. Several difficulties are being addressed. The first arises from the rapid increase in complexity when moving from one to two sentences and so on because premises and concepts can mesh. Currently, we are attempting to enable the database to develop all the possible correct structures. Then marking will be rapid because a corresponding tree needs to be matched rather than developed. The SE develops the student’s solution structure as he/she enters the solution. This can make solving and marking slow. At this stage, a more serious problem is that the student-editor cracks up when it has several disjoint, unrecognized links and concepts. This is frequently experienced as a student manipulates concepts in his solution. One fairly successful attempt to address this has been to delay marking until the student indicates by clicking a button that he/she is now ready to have his solution assessed. 11. Conclusions One of the most significant advantages of the response approach is that the students' understanding of the problem can be assessed before they even attempt the solution. A significant advantage of the program being written in Java is that the problem and solution structure can be down-loaded to the student’s machine. Students can enter answers and the program marks on-site. A real benefit we envisage is that marks will not be assigned for effort or out of sympathy for the student who is struggling. This is specifically one of the filters we intended to build in to the tool in order to get at the level of real understanding. To date, the development of the assessment tool has provided more insight than benefit in marking. So far the instructor and the student have both had to have a sophisticated top top understanding of the program and the subject material, which gives optimistically misleadingly results because the student does understand the problem and the solution well anyway! 12. References AMS, 1996, Views on high school mathematics education. Notices of the American Mathematical Society, 43(8):866–873. AMS, 1996b, Views on high school mathematics education. Notices of the American Mathematical Society, 43(9):980–986. AMS, 1997, Views on high school mathematics education. Notices of the American Mathematical Society, 44(2):197–206. top Disclaimer Articles published in SAJIM are the opinions of the authors and do not necessarily reflect the opinion of the Editor, Board, Publisher, Webmaster or the Rand Afrikaans University. The user hereby waives any claim he/she/they may have or acquire against the publisher, its suppliers, licensees and sub licensees and indemnifies all said persons from any claims, lawsuits, proceedings, costs, special, incidental, consequential or indirect damages, including damages for loss of profits, loss of business or downtime arising out of or relating to the user’s use of the Website. ISSN 1560-683X Published by InterWord Communications for the Centre for Research in Web-based Applications, Rand Afrikaans University