PRISM Journal PRISM Volume 4. Issue 1 (2022) https://doi.org/10.24377/prism.ljmu.0401213 https://doi.org/10.24377/prism.ljmu.0401213 41 © 2021 PRISM, ISSN: 2514-5347 Can graphical feedback from a ‘rainbow diagram’ help students reflect on the coherence of their academic writing? Stephen Foster,1 Denise Whitelock,2 Simon Cross,3 and Karen Kear,4 1 Institute of Educational Technology (IET), The Open University, (stephen.foster@open.ac.uk) 2 Institute of Educational Technology (IET), The Open University, (denise.whitelock@open.ac.uk) 3 Institute of Educational Technology (IET), The Open University, (simon.j.cross@open.ac.uk) 4 Faculty of Science, Technology, Engineering and Maths (STEM), The Open University (karen.kear@open.ac.uk) Received: 14/11/2020 Accepted for publication: 14/06/2021 Published: 03/11/2021 Abstract OpenEssayist is an automated writing evaluation system (AWE) designed to provide immediate textual and graphical feedback to students to help them improve their academic writing. One of the graphical visualisations as part of OpenEssayist, (named a ‘rainbow diagram’), illustrates how well the key concepts within the writing are connected. The rainbow diagram element has been subjected to research by Whitelock et al., (2014), who determined that participants could identify patterns across the diagrams, identifications which corresponded to essays awarded low-grade or high- grade marks by tutors. The research reported as part of this paper is a follow-on study, developed to determine how participants might use the rainbow diagram to improve academic writing. Thirteen (n=13) PhD students were interviewed face-to-face whilst an eye-tracker recorded their gaze on a rainbow diagram produced from an example of their own writing. The current work confirms that students can use rainbow diagrams to identify content that corresponds to high-grade and low-grade work in essay writing. Building on Whitelock’s research, the study also shows that the rainbow diagram can be used further, to enable students to understand coherence and structure within academic writing, and to facilitate reflection on what actions should be taken to improve their writing. Keywords: automated writing evaluation, OpenEssayist, rainbow diagram, student academic writing, writing skills 1. Introduction Academic writing can be challenging as it involves the development of ideas and the translation of those ideas into text, in ways which conform to the style and convention of a student’s subject area (Torrance et al., 1994). Related to this, students require support to improve academic writing skills; furthermore, surveys https://doi.org/10.24377/prism.ljmu.0401213 https://doi.org/10.24377/prism.ljmu.0401213 https://openjournals.ljmu.ac.uk/index.php/prism/ mailto:stephen.foster@open.ac.uk mailto:denise.whitelock@open.ac.uk mailto:simon.j.cross@open.ac.uk mailto:karen.kear@open.ac.uk https://orcid.org/0000-0002-9897-4975 https://orcid.org/0000-0001-8741-8503 PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 42 4(1) often express student dissatisfaction with the type of assignment feedback that they receive (Nicol, 2010). However, the changed economies of scale within higher education, due to increased student numbers, mean that tutors are generally unlikely to have contact time available to navigate and respond to these expectations (Field et al., 2013). One alternative can be to provide automated feedback generated independently of the tutor. However, as Stevenson & Phakiti (2014) point out, there is only modest evidence available to suggest that automated feedback can improve students’ writing. This study seeks to add to that evidence, by looking at whether a new form of automated graphical visualisation – showing the coherence and structure of academic writing – can provide feedback to help students reflect on ways that writing might be improved. Interpretation of this graphic feedback, (called a ‘rainbow diagram’), requires students to identify compactness and graphic outliers1. To establish where students looked whilst doing this, an eye- tracker was used. Assignment feedback in higher education traditionally consists of comments provided by a tutor, for the student to read and retrospectively reflect on (Nicol, 2020). Nicol, (2010) observes that research suggests a need to improve the quality of this type of feedback. Nonetheless, whether comments are given formatively, as a student is drafting, or summatively, after the writing has been submitted, comments alone do not lead to improvement. A student needs to compare their work with the comments and generate internal feedback (Nicol, 2019), from which they can then make improvements as part of their approach to writing (Nicol & Macfarlane‐Dick, 2006). Thus feedback can be seen as a dialogue between teacher and student where learning is constructed through a cycle of feedback and reflection (Askew & Lodge, 2000). Research shows that the growth in student numbers presents a challenge to the provision of tutor feedback. Furthermore, surveys of students consistently find that satisfaction with feedback provision ranks lower than other features of their 1 Please see page 3 ‘What is a rainbow diagram?’ for an explanation of compactness and graphic outliers. course (Nicol, 2010). With this in mind, feedback from an automated writing evaluation (AWE) system might be able to help, by providing the necessary stimulus for students to constructively and proactively engage with feedback on their writing. AWE systems are a form of ‘learning system’, the development of which has been facilitated by advances in natural language processing (NLP). The ability of a computer program to interpret the meaning of text now facilitates a more complex analysis of language, which in turn can enhance the automated marking of essay assignments (Shermis & Burstein, 2013). Indeed NLP has been identified as one of the most successful methods for analysing writing (Shum et al., 2016). Consequently, many different AWE systems have been developed, such as Criterion (Burstein et al., 2003); WriteToLearn (Landauer, 2003); and, OpenEssayist (Field et al., 2013). One of the challenges of AWE development is establishing meaningful and accessible ways of displaying complex NLP data in a user-friendly way. Ware (2013) identifies that a good way of comprehending such complex data is through visualisations. While Whitelock et al., (2014) observe that such comprehension can occur through pattern identification, users must also understand the patterns within the context of the task that they are undertaking. Thus, when visualisations are used to provide feedback on academic writing it is important that users can not only interpret the patterns but also use them to improve their work. Picking up on this, my work here reports the findings of a small study which set out to investigate whether the graphical visualisation feedback within OpenEssayist (the rainbow diagram) can be interpreted by students in ways that facilitate reflection on the coherence of their academic writing, with a view to improving it. OpenEssayist OpenEssayist is an automated formative feedback system designed to provide feedback to help students improve their academic writing when no other PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 43 4(1) feedback is available. The feedback is intended to encourage students to reflect on the content and structure of their writing. OpenEssayist does this by providing feedback on essay content and structure through text and graphic visualisations (Whitelock et al., 2013). Following reflection, a student would re- draft the essay as required and then re-submit it to OpenEssayist to obtain further feedback. The cycle of feedback and submission continues until the student is confident that their draft meets the assignment criteria and submits it for summative assessment (Whitelock et al., 2015). Therefore OpenEssayist, instead of suggesting detailed correction for an essay, facilitates reflection on what has been written by the writer (Whitelock, 2018). This self-evaluative approach accords with what Beaumont et al., (2011) see as the fundamental aim of feedback, the development of a student’s capacity for self- regulated learning. Within OpenEssayist a new type of graphic feedback is offered, called a rainbow diagram; this helps the reflective process by showing how well key concepts within the essay are connected. The rainbow diagram has been the subject of research by Whitelock et al., (2014) who found that participants could determine – through comparison of the diagrams – essays that were awarded low-grade or high-grade marks by tutors. As such my paper here details a follow-on study developed to enhance Whitelock et al.’s research, to further explore the ways in which participants might use the rainbow diagram to improve their own academic writing. What is a Rainbow Diagram? The rainbow diagram is a graph which provides a visualisation of the interconnectedness of sentences within an essay, thereby providing an overall indication of essay coherence (see Figures 1 and 2 below). Each node (or dot) in the rainbow diagram represents a sentence within the essay which has some relevant words in common with at least two other sentences (Whitelock et al., 2014). The node which represents the first sentence in the essay will be violet and the node which represents the last sentence will be red. The nodes in-between change through the colours of the rainbow such that sentences towards the beginning of the essay will be shades of violet and sentences towards the end of the essay will be shades of red. An algorithm places a connecting line between one node and another node when the same relevant word appears in the sentence represented by each node (Whitelock et al., 2014). The algorithm compares the sentences with each other and derives a value representing their interconnectivity. That value determines how close the node representing one sentence is, in the diagram, to the node representing the other, linked, sentence (Whitelock et al., 2014). In a well-structured essay the nodes will be close to each other and nodes of similar colours tend to be grouped near each other towards the centre of the diagram (see Figure 1). This is because the sentences associated with the introduction (violet nodes) are grouped near to the nodes associate with the conclusion (red nodes) (Whitelock et al., 2014). In a less well-structured essay the nodes will tend to be more dispersed with the red nodes towards the outside of the diagram and the violet nodes toward the centre (see Figure 2). Early drafts of an essay might have more dispersed nodes. Each time an essay is redrafted and refined the nodes in the rainbow diagram should become more central, thereby suggesting the drafting has improved structure and coherence of the essay (Whitelock et al., 2014). Figure 1 High-grade essay PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 44 4(1) Figure 2 Low-grade essay What is Eye-Tracking? This study made use of Tobii eye-tracking equipment situated in a laboratory at The Open University. Eye-tracking devices can provide a dynamic record of where an individual looks on a computer screen (Poole & Ball, 2006) and are widely used to record eye-dwell time on areas of interest (Hogarth et al., 2008). Two eye-tracker data outputs were used, the ‘gaze plot’ and ‘heat map’. The gaze plot is a visualisation which shows which points on a visualisation are viewed, the order in which they are viewed and for how long they are viewed (see Figure 3). It is a dynamic plot and builds up as a participant views the screen. Each gaze point is represented by a circle and the larger the circle the longer the gaze time. The heat map is also a dynamic visualisation. It uses colouration to show where on the rainbow diagram a participant gazed most (see Figure 4) and thus identifies what most drew a participant’s attention. Figure 3 Example of a gaze plot Figure 4 Example of a heat map Eye-tracking can be used to determine where participants look during the interpretation of graphics (Mayer, 2010). To interpret the rainbow diagram, students had to identify which nodes were well connected and which were not. Use of the eye- tracker produced a record of where participants looked when undertaking this process. The data gathered was also used as stimulus for participants’ verbal reflection on why they looked where they did. Research Questions The study objective was to explore whether the rainbow diagram could help students reflect on the PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 45 4(1) coherence of their writing. It sought to do this through three research questions (RQ): RQ1: Can graphical feedback from a rainbow diagram help students reflect on the coherence of their academic writing? RQ2: What barriers do students perceive to using the rainbow diagram? RQ3: Can an eye-tracker help understanding of where students’ look on a rainbow diagram? 2. Method Participants were drawn from full-time PhD students at the Centre for Research in Education and Educational Technology (CREET) at the Open University. Invitations were sent to 47 students, of whom 13 agreed to participate in the research. To maintain anonymity students are referred to by research participant numbers, for example RP 23. The research method consisted of three parts. Part 1 introduced students to the principals of the rainbow diagram through a paper-based exercise. After reading a brief introduction to the rainbow diagram participants were provided with a folder containing four sections. Each section contained examples of four rainbow diagrams of the same grade: •Section 1 - high-grade essays •Section 2 – low-grade essays •Section 3 – medium-grade essays • Section 4 - Stanford Booth prize essays (i.e. very good essays). The rainbow diagrams used for the high, medium and low-grade essay examples, were from essays used for research undertaken by Whitelock, et al., (2014). They were written by participants drawn from a subject panel2 maintained by The Open University’s Department of Psychology. The Stanford Booth prize competition essays were drawn from essays submitted for consideration in the competitions held in 2006 and 2007. The length of the Stanford Booth 2 The ‘subject panel’ was a list of volunteers who had indicated in advance that they were happy to be approached to participate in research. prize essays was slightly reduced to comply with the word limit for submission to OpenEssayist. The coherence of the Stanford Booth prize essays was such that despite the reduction in length, the rainbow diagrams produced were densely connected with a core of overlapping nodes. In part 1 participants were asked to use what they had learnt from the introductory document to identify which section within the folder contained examples illustrating which type of essay. They were provided with written instructions for completing the exercise and an answer grid (see Figure 5). Figure 5 Part 1 exercise answer grid Table 1 details the rating criteria for the rainbow diagram used by the participants. The criteria were the same as that used by Whitelock et al., (2018). Low-scoring diagrams Medium-scoring diagrams High-scoring diagram Not densely connected Densely connected area but some outlying nodes Densely connected Red nodes (conclusion) not central Red (conclusion) and violet (introduction) not so closely connected Red nodes (conclusion) central Few links between violet (introduction) and red (conclusion) nodes Close links between violet (introduction) and red (conclusion) nodes PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 46 4(1) Table 1 Rating criteria for rainbow diagrams (Whitelock et al, 2018) Whilst deciding, students were asked to ‘think aloud’ (van Someren et al., 1994) and explain their thought processes for determining which set of rainbow diagrams represented which category of essay. Semi-structured questions were used to clarify and probe their reasoning. For part 2 of the research, students provided a piece of their own academic writing which was copied into OpenEssayist to produce a rainbow diagram. Students were asked to view the rainbow diagram of their work and used a ‘think aloud’ protocol to explain how they interpreted the feedback and how they might use it to improve their work. If required, the student’s explanation and reasoning was probed and clarified using semi-structured questions. The rainbow diagram was displayed on a computer monitor fitted with an eye -tracker. During viewing of the rainbow diagram, the eye-tracker recorded where on the diagram the students’ gazed, how long they gazed for and the track they took from one gaze point to another. The eye-tracking data enabled triangulation (Cohen et al., 2011) of interview data and facilitated richer recall data in part 3 of the study. Part 3 of the interview was a period of stimulated recall (Gass & MacKey, 2000) when the eye-tracking gaze plot and heat map were shown to the student. Through contextual semi-structured interviews, students were asked to recall their thoughts as to why their gaze went to the points on the rainbow diagram that it did and what they were thinking at that time. Students were also asked semi-structured questions about the usefulness to them of the rainbow diagram for reviewing their academic writing. The interview data from the research was analysed using Braun and Clarke’s (2006) six phase thematic analysis process: data familiarisation; coding; generation of themes; reviewing themes; defining and naming themes; writing up. Coding followed Braun and Clarke’s (2019 p.594) inductive approach and was based on the researcher’s interpretation of the semantic content of the data. It was undertaken using NVivo software. Subsequently the codes were grouped into themes. The part 1 exercise sought data for RQ 1 and 2. The part 2 and 3 exercises sought data for RQ 1-3. 3. Findings and Discussion This section presents and discusses the research findings. It identifies the outcome of the part 1 exercise, introduces five themes which emerged from a thematic analysis of the interview data from all three parts of the study and then discusses the eye- tracking data and how it triangulated the interview data. Part 1 of the research followed a pattern similar to that of Whitelock et al., (2014): students were asked to identify which section within a folder represented which of the high, medium, low and Stanford Booth Prize essay types. Of the 13 participants, 11 correctly identified which section in the folder represented which type of essay (see Table 2). Participants Correct Responses for Section 1 Correct Responses for Section 2 Correct Responses for Section 3 Correct Responses for Section 4 N=13 11 13 13 11 Table 2 Student responses to the identification exercise for the four different types of rainbow diagrams Individual participant responses are shown in Table 3 (Figure 5 [above] details the letter meanings). RP No. Section 1 Section 2 Section 3 Section 4 23 B D C A 24 B D C A 25 B D C A 26 Undecided D C Undecided 27 B D C A 28 B D C A 29 B D C A 30 A D C B 31 B D C A 32 B D C A PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 47 4(1) 33 B D C A 34 B D C A 35 B D C A Table 3 Participant responses for the part 1 exercise Students were asked to ‘think aloud’ whilst conducting the exercise and to explain their reasoning for making the choices they did. Participants tended to make visual comparisons between the rainbow diagrams to determine which ones were the most densely clustered and therefore represented the higher-grade essays. Some participants found it more challenging than others to differentiate the medium grade essays from the high-grade essays, although all students did this. Two students were unable to distinguish the high-grade essays from the Stanford Booth prize essays, as the rainbow diagram pattern for both essay types was tightly clustered. This result differs from that of Whitelock et al., (2014) where all participants correctly determined which rainbow diagrams related to which type of essay. However, as the Stanford Booth prize essays are also ‘high-grade essays’, it is perhaps not surprising that two participants did not distinguish between them. Thematic analysis of all three parts of the research data identified five themes: •Comparison interpretation •Writing confidence •Academic writing skills •Comprehension of the rainbow diagram •Barriers and openings Each of these themes will now be briefly discussed. Theme - Comparison Interpretation During part 1 of the research the ‘think aloud’ protocol identified that most students made comparisons between the rainbow diagrams in the different sections of the exercise folder. For example, RP.27 said, ‘So I can see that most of them were in the middle, central they are central, and then I compared them with the folders, the essays of the folder group three, they were in the centre but not that close to the violet nodes’. The theme suggested that from reading a short one-page introductory document it was possible for students to understand which pattern of rainbow diagram represented which type of essay. Students also made comparisons during part 2 of the study. A common pattern was for participants to think back to part 1 and make use of the knowledge about what the different node clusterings on a rainbow diagram might mean. For instance, RP.25 said, ‘Oh, it looks like, following the previous instructions, a kind of a medium because you have quite a lot of red dots in the middle, some of them are spread around, and the violets are near the middle’. From these comments, it appeared RP.25 was using the knowledge gained from the part 1 exercise to interpret their own academic writing. RP.30 commented that: ‘It's about what I expected to see, there is very little connection between the beginning and the end though they are moderately grouped. Obviously if you are believing the premise of the software you should be driving everything towards the centre and linking your intro to your conclusions and your [main] body and making all the dots pile up on top of each other’. RP.30’s comments inferred they were able to visualise what they thought a rainbow diagram of their work would look like having completed the part 1 exercise. During part 3 of the research, participants reviewed the eye-tracking heat map and gaze plot whilst commenting on why they had been looking where they did on the rainbow diagram. For example, RP.28 commented: ‘I mostly look at the centre because I was trying to identify how many red spots I have and how many violets spots I have, if they are close to each PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 48 4(1) other and if there are a lot of links between them, so yes that's the main reason’. This suggested that students, having read a short one-page introduction to the rainbow diagram, were following the instructions provided to interpret it. It also suggested that students were using the knowledge from the part 1 exercise to interpret the rainbow diagram of their own writing. Theme - Writing Confidence There were several areas in which use of the rainbow diagram feedback might be able to improve student confidence in the coherence of their academic writing. For six of the 13 students English was not their first language. Several of these participants suggested that the rainbow diagram could give confidence they had written a coherent piece of work in English. RP.28 commented: ‘I think it could be really useful for my writing. Not being a native speaker that's one of the things that I'm struggling with. Writing is one of the things that I find difficult’. RP.28 was asked if they thought there might be some benefit to students with English as a 2nd language in using a tool like OpenEssayist. They replied: ‘Yes, definitely yes because, so academic writing in […] is quite different with, actually it's totally different with academic writing in English. In […] academic writing has to do with very very long and philosophical sentences while in English you need to be very precise, very short’. The data suggested that students whose first language is English can also gain confidence in the coherence of their writing from the rainbow diagram. For instance, RP.24 commented: ‘based on the diagrams that I saw in the previous one as well, like I'm happy it would be within the top two categories of, that we were shown in comparison so yeah, with those as my point of reference I am quite happy with the way that this essay turned out’. Some participants however, had doubts about the extent to which the rainbow diagram could give confidence in the cohesiveness of their writing. RP.33 said: ‘and I'm sure once you get used to looking at these things you might feel better about it but as an instant piece of, it is something that needs to whack you in the face, then it's not doing it for me yet’. Thus they acknowledged that with further experience they might gain confidence from rainbow diagram feedback, but they were not there yet. This comment is understandable, as the participant’s knowledge of the rainbow diagram was limited to what they had learnt in the part 1 exercise. RP.32 suggested that the rainbow diagram might be useful for developing the confidence of 1st and perhaps 2nd year undergraduate students who were learning academic writing. They said: ‘if you are just starting to learn how to write essays I think you should definitely start with learning about the structure and it’s also easy to write about anything in the world, having the rainbow in your mind you remember that you need to stay on topic and make sure that every paragraph is linked to what you said before’. Theme – Academic Writing Skills This theme pulls together a range of potential benefits for academic writing of using the rainbow diagram. One potential benefit is assistance to students with learning disabilities. RP.34 commented that: ‘people with learning disabilities, if there is a common thread that they can follow that is going to make it 10 times easier for them and this I think helps with that, which is quite interesting, not sure if it's anything they ever thought of using before for accessibility.’ However it is acknowledged that, particularly for students who have colour vision impairment, the rainbow diagram in its current format will present some challenges. One objective of OpenEssayist PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 49 4(1) feedback is to help students to become self-regulated learners. This was acknowledged by RP.27 who said: ‘Okay I can see the rainbow diagram as a tool for self-assessment where it's not the teacher who's telling me you did this, look at that, but it's me more making some inferences looking at the diagram by my own means and trying to work it out then if I make sense of it’. Echoing this RP.34 said: ‘I think it's such a good tool to have even like, even for at our level where we are writing a thesis. I'm sure it just makes it so much easier to read the, for the reader, it makes more sense as you are going through’. Participants who saw themselves as visual learners commented favourably on the rainbow diagram feedback. For example, RP.27 stated: ‘for me I always consider myself more visual, so when I see a graph that represents the structure of my essay how do sentences like meaning and semantics align with the structure [...] [it] is a very good advantage’. Theme – Comprehension of Rainbow Diagram. The fourth group of codes came together under the theme of understanding and using the rainbow diagram. Some participants had mixed views on how easy the rainbow diagram is to understand. For example, RP.35 commented: ‘think in terms of how busy the diagram is and it's quite easy to interpret, the closer together nodes the more coherent your writing is so that is pretty straightforward’. However, they also went on to say that: ‘In terms of the different colours and connections between the dots I'm not quite sure’. Thus while it might be fairly straightforward to predict whether an essay is of a high, medium or low- grade based on the closeness of the nodes, and also thereby assess the cohesiveness of the essay, it is perhaps more challenging to interpret the connections of colours between violet and red and what these mean for the coherence of the essay where, for example, there is a green shaded outlier node on the diagram. Perhaps unsurprisingly therefore, most students commented that they wished to be able to click on a node and receive information about which sentence within their writing the node represented. Several students pointed out that users would need a level of training to understand and use the rainbow diagram to improve academic writing. RP.24 pointed out that it would be important to train users on how to interpret the patterns. They commented: ‘I think it's just a different tool, as long as you know how to interpret it it's fine’. RP.29 similarly identified the need for training, suggesting that: ‘I think if people had that prior training that I have just done then they could look at that and they could really draw some beneficial conclusions from it’. Thus, where participants found it challenging to distinguish between some of the rainbow diagram graphs, they thought this was something which could be overcome with training and experience of use. Theme – Barriers and Openings Not all participants thought that the rainbow diagram feedback would assist them. For example, RP.26 commented that: ‘I don't think I'd find it particularly useful, I think that's just me really. I don't find it very intuitive in terms of how to interpret it’. However, they did also go on to say: ‘I think if you saw it as an animated effect, rather than a static image at the end, so you saw how your argument was being built up […] you can see that you are making the connections’. To some extent this observation reflects a limitation in the way part 2 of the study was PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 50 4(1) constructed, as participants reviewed work they had already completed. OpenEssayist does facilitate the viewing of each draft as a rainbow diagram as it is completed and permits a comparison between the rainbow diagrams of each draft. This enables users to see that they are making the connections required within the work. To improve the feedback, RP.33 commented that they would like to have a percentage score for the cohesiveness of the rainbow diagram graph, with 100% for a perfectly interconnected and cohesive graph. They said: ‘what I'd really like the rainbow diagram to do is give me a percentage, give me a mark or something because this is a whole like mass of interconnections, and it's not very interpretable […] so what I wanted to say, okay based on all of these criteria you have scored 89% in your rainbow diagram or something like’. Overall participants thought that the rainbow diagram provided a good visual clue as to the cohesiveness of their academic writing. Furthermore, they showed an understanding of how they could improve their writing using the rainbow diagram. Importantly, the outlying nodes of the rainbow diagram encouraged participants to reflect on their writing to identify the lack of interconnectedness which had caused the outlier, which they felt would facilitate an improvement in their writing. Whilst participants were generally able to interpret the rainbow diagram feedback and use it to make some suggestions as to how they might improve their academic writing, they also viewed the rainbow diagram critically and made suggestions as to what would improve it for them. Eye-Tracking Data Findings Whilst undertaking part 2 of the study students had their eye movement recorded by an eye-tracker. This provided a visual indication of where participants were looking to compare with what they said about where they were looking whilst ‘thinking aloud’. The heat maps universally showed that students’ attention was attracted most to the centre of the rainbow diagram (see Figures 6 and 7 [below] for examples). Figure 6 Heat map for RP 29 Figure 7 Heat map for RP 32 The gaze plots showed that students tended to initially look at the centre of the rainbow diagram. Their gaze then moved to look at an outlier node before moving to look back to the centre of the rainbow diagram. This pattern then tended to be repeated for other outlier nodes. The interviews confirmed that participants looked at the centre of the rainbow diagram to assess its compactness, then the outlier nodes to assess the interconnectedness of their work. For example, RP.23 said: ‘So I think I started looking at the central bit because I thought it was quite dense then I started looking at the outer elements of where there are kind of a few nodes on the outside of things, I was probably there trying to fix on the connections PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 51 4(1) about how strong they were and where they were linking to’. The heat maps, along with the gaze plots and participant interviews, evidence that the rainbow diagrams were interpreted systematically by participants. They looked at the centre of the rainbow diagram to determine how tightly clustered the nodes were, and therefore how cohesive their writing was. They then looked at the outlier nodes and their connections to the central nodes to determine how connected the outlier nodes were and decide what changes were required to make their writing more cohesive. An element of that process involved determining, via the colour of the node, where in their writing the sentences represented by outlier nodes were. 4. Summary of Findings The part 1 exercise demonstrated that students were able to distinguish between high, medium and low-grade essays. Though two participants did not distinguish the Stanford Booth prize essays from high grade essays, overall the result of the exercise reflected the findings of Whitelock et al., (2014) that students are able to use the rainbow diagram to distinguish between the essay types. Part 2 of the study added to the research of Whitelock et al., (2014) and showed that students can use their understanding of a rainbow diagram to review the coherence and structure of their writing and reflect on how they might improve it. How participants described their review of the rainbow diagram was triangulated by eye-tracking data, which was discussed with participants in part 3 of the study. This showed that participants tended to review the rainbow diagram systematically, their gaze going from the centre to outlier nodes then back to centre. The study answered RQ 1 by showing that rainbow diagram feedback can help students reflect on the structure and coherence of their writing. It enabled students to recognise when their writing was less well constructed, consider how it might be improved and identify what an improvement might look like on a rainbow diagram. The rainbow diagram therefore has the potential to provide feedback to improve academic writing and give confidence that writing is well written. In answer to RQ 2, the study identified no substantial barriers to using the rainbow diagram. After reading a brief introductory document, students were able to understand the graphic and use it to interpret their writing, though one student did state they did not find the rainbow diagram particularly intuitive and made suggestions as to how it could be improved for them. The study answered RQ 3 by showing that eye-tracking can reveal where students looked on the rainbow diagram. The data triangulated students’ verbal explanations and facilitated stimulated recall as to why students looked where they did. 5. Study Limitations and Further Research There are several potential limitations to the findings of this study. A possible limitation of part 1 was that the printed rainbow diagrams of the Stanford Booth prize essays were produced from an electronic source, whereas the other three essay types were photocopies of previously printed rainbow diagrams. The difference was evident as the electronic source of the Stanford Booth prize essays produced a clearer print. This could possibly suggest those rainbow diagrams came from a different source. Several participants commented on difference in print clarity. However as, with two exceptions, all students commented on extreme compactness of the Stanford Booth prize essay rainbow diagrams, it is assessed that the effect of the print clarity on the result was negligible. Part 2 of the research indicated students were able to interpret the rainbow diagram and use it to make suggestions as to how they might improve their academic writing. A limitation of this finding is that students were commenting on academic writing which they had already written, whereas the rainbow diagram feedback is designed to help students at the drafting stage of writing. The participants did not, therefore, interpret their writing in the true context of automated formative feedback. Furthermore, their knowledge of the academic writing submitted might PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 52 4(1) be dated and not readily recalled. Indeed, a potential limitation of the think-aloud method is that only information which enters a person’s short-term memory can be processed verbally (Ericsson and Simon, 1980). Nonetheless students did analyse their writing using the rainbow diagram and made suggestions as to how it could be used to improve it. Further research will obtain data from students who have used the rainbow diagram whilst drafting an assignment and explore the extent to which the rainbow diagram can provide feedback on different types of academic writing. Such data will provide a more robust evaluation of the usefulness of the rainbow diagram to students. 6. Conclusion This study has followed on from the research of Whitelock et al., (2014) and explored how students might use the rainbow diagram to improve their academic writing. Thematic analysis (Braun and Clarke, 2006) was used to examine the data and five themes were identified from this process. Overall analysis of the data from the study showed that the rainbow diagram can give students an understanding of the coherence and structure of their academic writing. Through that understanding students can use knowledge of their work to reflect on what actions, if any, need to be taken to improve the coherence and structure of it. Undertaking this process can give students confidence in their writing. To facilitate use of the rainbow diagram, students were given instruction on how to interpret it. This was important, as all participants made some form of comparative reference between rainbow diagrams to help with their interpretation. Participants were not wholly uncritical of the rainbow diagram. It was sometimes challenging to clearly distinguish a medium-grade essay from a high- grade or low-grade essay, though all participants did so. Whilst most participants were confident in interpreting the connections between the violet (introductory) and red (concluding) nodes, they were less confident in interpreting what it meant if intermediate-coloured nodes, from the main body of their work, were outliers from the centre of the rainbow diagram. Following on from this, most participants suggested that it would be useful to be able to place the computer cursor over a node and find out which sentence the node related to. Overall participants concluded that the rainbow diagram can help provide confidence in the structure and coherence of their academic writing and facilitate reflection on how structure and coherence might be improved. PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 53 4(1) 7. Disclosure statement The author(s) declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. 8. Acknowledgements The research was funded by The Open University. The authors would like to thank the students who took part in the study. 9. Open Access Policy This journal provides immediate open access to its content with no submission or publications fees. This journal article is published under the following Creative Commons Licence: This licence allows others to read, download, copy, distribute, print, search, or link to this article (and other works in this journal), and/or to use them for any other lawful purpose in accordance with the licence. PRISM is also indexed in the world largest open- access database: DOAJ (the Directory of Open Access Journals). DOAJ is a community-curated online directory that indexes and provides access to high quality, open access, peer-reviewed journals. https://doaj.org/toc/2514-5347?source=%7B%22query%22%3A%7B%22filtered%22%3A%7B%22filter%22%3A%7B%22bool%22%3A%7B%22must%22%3A%5B%7B%22terms%22%3A%7B%22index.issn.exact%22%3A%5B%222514-5347%22%5D%7D%7D%2C%7B%22term%22%3A%7B%22_type%22%3A%22article%22%7D%7D%5D%7D%7D%2C%22query%22%3A%7B%22match_all%22%3A%7B%7D%7D%7D%7D%2C%22size%22%3A100%2C%22_source%22%3A%7B%7D%7D https://doaj.org/toc/2514-5347?source=%7B%22query%22%3A%7B%22filtered%22%3A%7B%22filter%22%3A%7B%22bool%22%3A%7B%22must%22%3A%5B%7B%22terms%22%3A%7B%22index.issn.exact%22%3A%5B%222514-5347%22%5D%7D%7D%2C%7B%22term%22%3A%7B%22_type%22%3A%22article%22%7D%7D%5D%7D%7D%2C%22query%22%3A%7B%22match_all%22%3A%7B%7D%7D%7D%7D%2C%22size%22%3A100%2C%22_source%22%3A%7B%7D%7D https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doaj.org/ PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 54 4(1) 10. References Askew, S., & Lodge, C. (2000). Gifts, ping-pong and loops—Linking feedback and learning. In Feedback for learning. Routledge. Beaumont, C., O’Doherty, M., & Shannon, L. (2011). Reconceptualising assessment feedback: A key to improving student learning? Studies in Higher Education, 36(6), 671–687. https://doi.org/10.1080/03075071003731135 Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806 Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays. IAAI, 3–10. Cohen, L., Manion, L., & Morrison, K. (2011). Research Methods in Education. Routledge. Ericsson, K. A., & Simon, H. A. (1980). Verbal Reports as Data. Psychological Review, 87, 215–251. Field, D., Pulman, S., Van Labeke, N., Whitelock, D., & Richardson, J. T. (2013). Did I really mean that? Applying automatic summarisation techniques to formative feedback. Association for Computational Linguistics Proceedings, 277–284. Gass, S., & MacKey, A. (2000). Stimulated Recall Methodology in Second Language Research. Lawrence Erlbaum Associates. Hogarth, L., Dickinson, A., Austin, A., Brown, C., & Duka, T. (2008). Attention and expectation in human predictive learning: The role of uncertainty. Quarterly Journal of Experimental Psychology, 61(11), 1658–1668. https://doi.org/10.1080/17470210701643439 Landauer, T. K. (2003). Automatic Essay Assessment. Assessment in Education: Principles, Policy & Practice, 10(3), 295–308. https://doi.org/10.1080/0969594032000148154 Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171. https://doi.org/10.1016/j.learninstruc.2009.02.012 Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501–517. https://doi.org/10.1080/02602931003786559 Nicol, D. (2019). Reconceptualising feedback as an internal not an external process. Italian Journal of Educational Research, Special Issue, 71–83. Nicol, D. (2020). The power of internal feedback: Exploiting natural comparison processes. Assessment & Evaluation in Higher Education, 1–23. https://doi.org/10.1080/02602938.2020.1823314 Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090 Poole, A., & Ball, L. J. (2006). Eye Tracking in Human- Computer Interaction and Usability Research: Current Status and Future Prospects. 14. Shermis, M., & Burstein, J. (2013). Handbook of Automated Essay Evaluation Current Applications and New Directions. Routledge. Shum, S. B., Knight, S., McNamara, D., Allen, L., Bektik, D., & Crossley, S. (2016). Critical perspectives on writing analytics. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16, 481–483. https://doi.org/10.1145/2883851.2883854 Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007 Torrance, M., Thomas, G. V., & Robinson, E. J. (1994). The writing strategies of graduate research students in the social sciences. Higher Education, 27(3), 379–392. https://doi.org/10.1007/BF03179901 van Someren, M., Barnard, Y., & Sandberg, J. (1994). The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. Academic Press. Ware, C. (2013). Information Visualization: Perception for Design. (3rd ed.). Elsevier. Whitelock, D., Field, D., Pulman, S., Richardson, J., & Van Labeke, N. (2013). OpenEssayist: An automated feedback system that supports university students as they write summative essays. The 1st International https://doi.org/10.1080/03075071003731135 https://doi.org/10.1080/2159676X.2019.1628806 https://doi.org/10.1080/17470210701643439 https://doi.org/10.1080/0969594032000148154 https://doi.org/10.1016/j.learninstruc.2009.02.012 https://doi.org/10.1080/02602931003786559 https://doi.org/10.1080/02602938.2020.1823314 https://doi.org/10.1080/03075070600572090 https://doi.org/10.1145/2883851.2883854 https://doi.org/10.1016/j.asw.2013.11.007 https://doi.org/10.1007/BF03179901 PRISM (2022) Foster, Whitelock, Cross & Kear (2022) PRISM 55 4(1) Conference on Open Learning: Role, Challenges and Aspirations. Whitelock, D., (2018). Advice for Action with Automatic Feedback Systems. In Software Data Engineering for Network eLearning Environments Analytics and Awareness Learning Services. Springer International Publishing AG. Whitelock, D., Field, D., Pulman, S., Richardson, J. T., & Van Labeke, N. (2014). Designing and testing visual representations of draft essays for higher education students. In: 2nd International Workshop on Discourse- Centric Learning Analytics, 4th Conference on Learning Analytics and Knowledge (LAK2014) [Online]. http://oro.open.ac.uk/41845/1/dcla14_whitelock_etal. pdf Whitelock, D., Twiner, A., Richardson, J. T. E., Field, D., & Pulman, S. (2018). What Does a ‘Good’ Essay Look Like? Rainbow Diagrams Representing Essay Quality. In E. Ras & A. E. Guerrero Roldán (Eds.), Technology Enhanced Assessment (Vol. 829, pp. 1–12). Springer International Publishing. https://doi.org/10.1007/978-3-319-97807- 9_1 Whitelock, D., Twiner, A., Richardson, J. T. E., Field, D., & Pulman, S. (2015). OpenEssayist: A supply and demand learning analytics tool for drafting academic essays. 208–212. https://doi.org/10.1145/2723576.2723599 http://oro.open.ac.uk/41845/1/dcla14_whitelock_etal.pdf http://oro.open.ac.uk/41845/1/dcla14_whitelock_etal.pdf https://doi.org/10.1007/978-3-319-97807-9_1 https://doi.org/10.1007/978-3-319-97807-9_1 https://doi.org/10.1145/2723576.2723599