Australasian Journal of Educational Technology, 2021, 37(3). 22 Towards faster feedback in higher education through digitally mediated dialogic loops Jill Willis, Andrew Gibson, Nick Kelly, Nerida Spina, Jennifer M. Azordegan, Leanne Crosswell Queensland University of Technology How feedback is understood and enacted has shifted from the traditional practice of providing individual feedback on summative tasks at key points to a more ongoing series of dialogues between the teacher and students during the teaching period. This paper reports on the experiences of designing faster feedback through weekly dialogic feedback loops to enhance students’ personal connection to their learning while providing teachers with faster, actionable feedback data to inform learning design. A pragmatic inquiry considered how benefits might potentially be amplified through the use of digital technologies. Data included student reflections collected via the GoingOK web application, interviews and focus groups. The findings identify and theorise four types of digitally mediated feedback loops: students in computer-mediated dialogue with themselves; students and teachers in dialogue with each other; the reflection on how feedback informed learning; and the sociotechnical dialogue informing ongoing technical design. Three design dilemmas that were experienced by teachers as they enacted digitally mediated dialogic feedback loops are articulated, alongside the principles that enabled responsive design. Understanding these design elements is fundamental if automation of some parts of the feedback loop through reflective writing analytics is to be considered both feasible and desirable. Implications for practice or policy:  Digitally mediated feedback loops can facilitate faster feedback, enabling students to reflect on their learning and providing teachers with access to new insights about diverse learners.  Feedback technology can challenge existing ideas about feedback.  Faster feedback can save teachers time, but efficiencies are likely to depend on an increased human workload in the short term as automation technologies can be slower to develop.  Sociotechnical innovation requires collective dialogue between educators and digital developers, across asynchronous timelines. Keywords: faster feedback, dialogic feedback, digital feedback, reflective writing, sociotechnical design Introduction Feedback is a high interest topic within higher education in Australia with promises of enhanced learning experiences for students. Yet higher education students report that the nature, timeliness and specificity of feedback is insufficient (Nash & Winstone, 2017). Traditional conceptions of feedback have focused on teachers delivering comments to students. These notions have been challenged by new paradigms that prioritise student sense-making and recognise the socially constructed nature of feedback within curriculum design (Winstone & Carless, 2020). Feedback in this more student-centred approach is defined as “a process in which learners make sense of information about their performance and use it to enhance the quality of their work or learning strategies” (Henderson et al., 2018, p. 16). Instead of being something provided at the end of assessment, feedback is positioned as a series of interactions between a teacher and student referred to as dialogic feedback (Beaumont et al., 2008). Dialogic feedback involves communication and shared sense-making where the involved participants think and reason together (Gravett & Petersen, 2002). When it occurs within multiple feedback loops or spirals of interaction it enriches short-term and long-term development across a cohort (Carless, 2019) and is associated with enhanced student engagement and self-regulation (Boud & Molloy, 2013). Dialogic feedback enables teachers and students to share emerging ideas, listen and respond to the learning in progress throughout the learning process (Hill & West, 2020). Through iterative loops of feedback, students Australasian Journal of Educational Technology, 2021, 37(3). 23 can apply their learning to immediate and longer-term learning and assessment, and teachers can connect with their students as they understand the emerging challenges for each cohort as they engage with new and complex concepts. Yet finding sustainable processes and allocating precious teaching time to completing multiple loops of feedback within a semester of study remains a practical challenge for educators. With large lecture and tutorial groups and minimal workload allocation, university teachers rarely have time for labour-intensive work of dialogic feedback involving listening and responding to individuals. Within higher education, multiple feedback loops need to feasible and manageable. Thus, there is a need for new teacher-student feedback designs that are faster to facilitate and that involve individuals and groups in manageable ways. This paper details an exploratory, pragmatic inquiry that considered how the pedagogic benefits of dialogic feedback might potentially be amplified through the use of digital technologies, to provide faster feedback. Digital technologies enabling and automating faster feedback In this paper we propose that digital technology has the potential to facilitate what we refer to as faster feedback, by making it possible for teachers to provide more timely dialogic feedback. The term faster feedback was inspired by the work of Henderson et al. (2017), whose study showed that students value the capacity of digital technology to allow them to receive “instant” or “ much faster” (p. 1575) feedback that enabled them to gauge their own learning progress. Feedback technologies that enable this near-instant feedback are diverse, including technologies such as online quizzes and in-class polling (Han & Finkelstein, 2013) as well as more complex game-based learning (Holbrey, 2020), clinical simulations (Bernaitis et al., 2018) and automated writing feedback (Shibani et al., 2017). Rapid feedback technologies can help learners to feel they are on the right track (Deeley, 2018), but feedback focused on predetermined responses may be of limited value for deep learning (Winstone & Carless, 2020) or for transfer between assessment tasks (Hill & West, 2020). This research aimed to investigate technology that could lead to feedback that was open, dialogic and ultimately faster for students and for teachers to inform ongoing learning. Two kinds of technology relevant to faster feedback were investigated: technology for enabling students to engage in feedback loops through reflective writing and technology to automate parts of the feedback loops through reflective writing analytics. Reflective writing is a way of externalising the metacognitive and self- regulatory processes at the heart of feedback experiences. Students think deeply about their learning and themselves as learners, benefiting their learning (Ryan, 2015) and wellbeing (Pennebaker & Seagal, 1999). Web application technology has been shown to enable postgraduate coursework students to record reflections on their learning (Gibson et. al., 2017). When students complete reflections online (rather than on paper), it allows for ease of transfer (digitally, overcoming distance) and collation (from many sources into a single source). Analysis of student reflections can yield insights about the learner and their experiences within the learning context, which in turn has the potential to assist teachers with the development of relevant and timely feedback to inform student learning and responsive action. Automation of reflective writing can occur through reflective writing analytics (RWA), a specialised sub- field of learning analytics that analyses reflective text for the purposes of providing feedback for learning (Gibson, 2017). Learning analytics in its basic form is the utilisation of technology-mediated data in the service of improvement of learning and learning-related activities (Gibson & Lang, 2018). With RWA, the analysis of reflective writing can be executed quickly through computation so that the knowledge work of analysis by hand can be minimised, or averted entirely, and hence potentially speed up the feedback process. Automation may also address the conceptual challenge raised by Yang and Carless (2013) to find ways that technology can “make feedback provision possible beyond the temporal-spatial confinements of the classroom…and create opportunities for collective learning and individual reflection” (p. 292). To develop digital innovation, alongside pedagogic innovation so that faster feedback could be implemented for immediate practical benefit, two research questions were devised: (1) In what ways can a sociotechnical approach contribute to the enabling of faster dialogic feedback loops within a higher education learning context? (2) What can be learned from a sociotechnical approach about the potential for automation of feedback? Australasian Journal of Educational Technology, 2021, 37(3). 24 Sociotechnical conception of faster feedback We conceived that the development of faster feedback for large cohorts of students would require developments in the automation of feedback alongside the enabling of feedback. In a recent literature review about technology for assessment, Brady et al. (2019) identified that there is a complex interaction of factors that influences the technological development for assessment, and there is a need for more digital research informed by pedagogy and staff experiences. A sociotechnical paradigm was used to frame our response to this challenge (Gibson, 2017). A sociotechnical – rather than merely technical – process is one which harnesses both technological and pedagogical resources to the task of providing faster feedback to students. In this conception, both the pedagogical and technological play to their strengths, with the pedagogical resources (such as teacher and student interactions and principles of dialogic feedback) adding meaning and learning value, and the technological enablers and automaticity adding speed and the capacity to scale to large numbers of students. The sociotechnical implementation of dialogic feedback loops in this study was based on students’ reflective writing (Figure 1). Figure 1. Sociotechnical design for faster feedback This paper examines the sociotechnical implementation process of establishing feedback loops, using the reflective writing digital software GoingOK (http://goingok.org) and the outcomes experienced for students and teachers. The sociotechnical interactions are theorised in this paper to propose principles to guide pedagogic and technical informed future digitally mediated dialogic feedback loops. Methods A pragmatic inquiry approach (Gibson & Lang, 2019) informed the design of the sociotechnical conceptualisation so it was anchored to the anticipated practical effects of effective faster feedback for the participating students. Abductive pragmatic inquiry (conceiving in terms of what might be) is well suited to transdisciplinary research as the foundation is provided by emergent practical effects in the implementation context rather than by predefined disciplinary knowledge. In a sense, the research is neither pedagogically nor technologically founded, but rather pedagogical and technological knowledge jointly interpenetrates (Wickson et al., 2006) the sociotechnical method without subduing it within disciplinary boundaries. http://goingok.org/ Australasian Journal of Educational Technology, 2021, 37(3). 25 Social context and participants Two cohorts of postgraduate master’s students (N = 402) used the GoingOK web application in two semesters at an Australian metropolitan university in 2018. The unit was compulsory for all students enrolled in a Master of Education course and required students to engage with a substantial amount of theoretical reading in the sociology of education. The majority of students were enrolled off-campus (91%) and part-time (73%), and many were completing their studies while maintaining full-time work, family and community commitments. Many students were education professionals who were returning to study, often after some decades. Students recorded reflections on their learning using GoingOK. Three teachers (lecturers) worked with us to receive support in analysing feedback in weekly loops. The lecturers then presented a weekly summary of reflections to the students and adapted curriculum plans where needed. Data were gathered from 3118 student reflections, focus group interviews with 12 students, 3 lecturer interviews, meeting notes and from automated visualisations of group data. All data was deidentified and represented with codes to indicate S for students and L for lecturer. Institutional ethics approval was obtained for the study. No personal or identifying information was stored. Technology for feedback: GoingOK The GoingOK web application software was created in 2012 to facilitate an improved feedback process with a small group of early career teachers (Willis et al., 2017). By September 2019, GoingOK had been used by over 2500 people who had written more than 14,000 reflections. The software has been developed on an ongoing basis since 2012, from minor improvements through to a complete rewrite of the software, as a result of pragmatic design collaborations with academics and student participants. The present study can be considered as one phase in this design collaboration. Figure 2. Example of a GoingOK visualisation of self-reported feeling over time, as seen by a participant, and interface for using a slider and text to reflect upon current feelings Australasian Journal of Educational Technology, 2021, 37(3). 26 Data sources Data was collected about the feedback loops during the pragmatic implementation of the study. Individual reflections: The students wrote personal reflections at least twice per week using GoingOK. This task was mandatory but ungraded (satisfactory/unsatisfactory). The summative assessment task required students to write an extended essay based on one of their reflections and their readings. Teachers provided broad prompts such as “What ideas have been prompted by your reading this week?” and “How are your assumptions being challenged or confirmed?” Lengths of reflections ranged from a single word (“OK”) to some over 1000 words long. Students typically reflected on core readings and subject matter content, while making connections to their own personal and professional histories. Students were reassured that the writing could be informal rather than scholarly to encourage responses that were genuine and allowed for deep thinking. Each time an individual student posted an entry into their personal online GoingOK journal they added to their plotline (Figure 2). Table 1 provides the numbers of students and reflections for each cohort. Students using GoingOK to provide at least one reflection totalled 365 (mean reflections = 8.5). A total of 37 students negotiated an alternative approach to writing their reflections. Reasons for an alternate approach included lack of access to the Internet caused by a natural disaster, computing difficulties or personal concerns about sharing their thoughts in an online environment. Table 1 Students participating in the study Group Students using GoingOK Reflections recorded Cohort A (Semester 1, 2018) 191 2125 Cohort B (Semester 2, 2018) 174 993 Total 365 3118 Collective dialogue: After teachers reviewed the deidentified individual reflections, they provided feedback to the whole student group. This enabled collective dialogue within the group and began when teachers responded to the group’s reflective data. To facilitate the teacher’s response to the group, two members of the research team read the reflections and identified patterns of what students were discerning and deliberating about in their learning (Archer, 2007). Figure 3 is an example of a weekly summary and suggestions for pedagogic prompts. This was also preparation for future automation as each week we discussed with the teachers those aspects of the summaries that were meaningful or helpful. This informed ongoing discussions about designing for automation. This pattern of collective dialogue led to a step towards automation, with the addition to the GoingOK software of a spreadsheet of the deidentified group reflections that could be generated any time by the teachers. The GoingOK technology deidentified the author reflections to protect student identity by using a pre-assigned GoingOK anonymous username. This digital collection of the group reflections into a spreadsheet enabled the teacher to easily access reflections for the whole cohort, with greater convenience for teachers than other forms of reflection such as post-it- note collections or individual student blog posts. Reflections on design decisions and experiences: Data was gathered through individual interviews with 3 teachers, 12 students and 2 researchers, conducted at the end of semester. Group meeting notes between the research team and teachers identified ongoing design decisions. The technology infrastructure was redesigned between the semesters to be more enabling. Visualisations of group data were the first steps in RWA automation and included an automatically generated spreadsheet, a topic report of frequent words from the text entries, and a contour map (Figure 4 of reflection points for the group. Records of technical changes and updates were recorded in the software. Australasian Journal of Educational Technology, 2021, 37(3). 27 Figure 3. Example of pedagogic resource where meaning and value are extracted manually from collective reflections Findings A key finding of the work was the identification of four types of dialogic feedback loops that needed to be managed when designing technologically mediated feedback to be faster. All four types of loops were integral to the overall implementation and occurred at different temporal scales. The types consisted of (a) internal dialogue within the reflexive individual, in which students provided digitally mediated feedback to themselves twice weekly; (b) collective dialogue, in which asynchronous digital feedback from students’ group data created dialogue loops with teachers; (c) reflection on dialogue, in which students and teachers identified how sociotechnical feedback informed learning; and (d) a sociotechnical design dialogue, in which designers of feedback pathways and teachers were in dialogue about the design of pedagogy and automation in monthly meetings. The outcomes of the first three of these loops occurred within this exploratory study. The fourth loop was not fully closed, with continued automation development extending beyond the project. Australasian Journal of Educational Technology, 2021, 37(3). 28 Feedback loop 1: Reflexive individual students in computer-mediated dialogue with themselves The first feedback loop occurred when students made sense of their learning through a reflexive internal dialogue as expressed in their weekly GoingOK entries. Reflexivity occurs when individuals, through their inner conversation, consider links to their personal and professional lives which can trigger subsequent dedication to courses of action (Archer, 2007). What follows is a reasonably typical example of a student reflection – in this case, centred on the topic of social class and education: I can see some links forming between stuff I think about [in my job] at school and the readings and lecture and tute [tutorial] information. There's a really hard dynamic between what [my] students should be doing and what they are capable of doing (in regards to behaviour as well as academics - and social!!) that I struggle with - as other students see it as unfair that some get away with not obeying the same rules... The ones I am particularly thinking of as I write this come from dysfunctional homes, so the disadvantage is real. (S9) Students often wrote about their ambitions, doubts, concerns and strategies for future action, as they developed new identities and narratives around being a postgraduate student. For example: Getting started with your first assignment (after 10 years of not doing any study) is quite a daunting task. Procrastination is definitely the enemy here but it's time to get down to business as I'm not a quitter and I'm determined to set a good example for my daughters and my two neighbours (12 years and 10 years) who don't have great role models for their parents but see what I am doing and I know that I am changing the way they see education. (S10) Students engaged in a digitally mediated dialogue with themselves, recording moments of learning, challenges, realisations and frustrations. The feedback that learners provided to themselves was part of an ongoing meaning-making process, where task performance was entangled with their learning performances, life histories, ambitions and concerns. As their reflective plotlines built up, students could recognise and interrogate their own patterns of what they had recorded, supporting metacognitive processes of thinking and learning, such as “It’s time to get down to business” and identity statements such as “I’m not a quitter”. Students immediately received self-feedback from the slider and their visual, growing plotline, an affordance of digital reflection that is not available in other forms of reflection, such as handwritten journals or paper exit tickets. This first feedback loop was therefore occurring as a form of private dialogic self- feedback to the individual learner, with learning signalled by reflexive epistemic statements such as “I am particularly thinking of as I write this” and “I know that I am”. As students deliberated in their writing, they were also creating evidence of their learning for further interrogation in their first summative assessment task for the subject. Feedback loop 2: Collective dialogue between students and teachers The second feedback loop connected students with their teacher(s) through group feedback enabled by the collation of digital reflections from the cohort that week. With approximately 500 reflections (usually two per student) each week, it would have been very difficult and time-consuming to collect, deidentify, collate and generate without the enabling technology. The analysis was conducted manually, in the form of a summary for the teaching team with pedagogic ideas that could then become a focus of the next teaching event (see Figure 3). Automation was planned on the basis of what teachers and students found valuable. The teachers shared examples of student reflective writing in weekly tutorials and acknowledged concerns and highlighted effective learning strategies. Typically, teachers spent 10 minutes at the beginning of their class sharing the patterns that seemed meaningful from the previous week, before engaging in discussion with students and providing reassuring stories about typical trajectories of learning in the discipline. Students could benchmark their own experiences against the emerging norms of the cohort. The discussion also enabled teachers to contextualise student concerns by pointing to what was still to come and to recommend sources of assistance. Australasian Journal of Educational Technology, 2021, 37(3). 29 The teachers facilitated collective dialogue through online chat, audio and video options; after seeking peer examples, students became increasingly comfortable with giving peers feedback, engaging in dialogue and asking questions of the group. As Orsmond et al. (2005) indicated, while written feedback can provide considered commentary, verbal feedback enables misunderstandings to be clarified quickly and for a negotiation of meaning to occur between teachers and students. A supportive culture of peer dialogue grew, with teachers attributing it to the collective feedback loop. This feedback loop was also dialogic because the teachers were learning from student reflections. Students who could not attend tutorials and instead watched recordings indicated that they sometimes felt isolated from discussions. In response to this feedback, the teachers subsequently made comments such as “welcome to those of you listening online. We know from your reflections that some of you are experiencing ...”. Comments about asynchronous students’ lack of connection were reduced following this pedagogical approach. Feedback loop 3: Reflection on dialogue (students and lecturers identifying how feedback informed learning) Researcher dialogues with teachers and students after the conclusion of the semester indicated that there were benefits from the enabling technology for faster pedagogic feedback loops. Teachers found out much earlier in the semester how students were understanding the assessment requirements and what the students were learning and worrying about: “Things that we might not even think about started to come up” (L3). The worries could then be addressed quickly: The feedback was just amazing, because we normally wouldn't know how students were tracking so early in the semester... It allows you to know... the issue that people are really worried about. Things that we might take for granted... Obviously one person's wondering about something, others probably are as well. And then it allows you to spr ingboard into some questions from people, that some of these might prompt. (L2) By sharing the group-level patterns from the deidentified GoingOK reflections with students, teachers were able to create new emotional connections between the students, the topics and the teaching team. The teachers commented that “it’s really, really helpful for them to know that they're not the only ones feeling that way. Everybody's pretty much in the same boat. I think that’s extremely reassuring” (L2). Teachers could build on and value student knowledge and experience and “use examples from their workplace in our teaching, so that the content is really relevant...if I can see someone's a VET trainer, I can provide examples around that area. I wouldn't know that otherwise” (L2). The teachers also felt they were “getting affirmation about what we were doing, and direction about what we needed to do from that regular GoingOK feedback” (L3). This element of co-constructing the learning with students created democratic and participatory power relationships. The teachers noted an improvement in student learning, evident in the reflections and in the summative assessment as students applied the theoretical concepts more comprehensively, saying “I haven't ever come across a way of being able to access that level of thinking from students, even the ones who are there in your classroom” (L2). The weekly discipline of reflection and discussion meant that dialogic feedback was happening at a “capillary level” (L3) rather than just midway through the unit. The faster feedback was time efficient. Issues were identified and addressed early as a group, instead of multiple individual student emails: “I feel like it saves time. We support students. We have a better relationship”(L2). Automation would need to be developed further to sustain the timesaving benefit as the teachers were not able to read all reflections each week. Students mentioned learning and self-regulation benefits in interviews conducted after the semester. Reflecting supported their deeper thinking and moving beyond a focus on learning content: “after I learned the content and I wrote the reflection, I start to think more” (S3). Reflecting also facilitated the application of learning to future scenarios: “to try and think through those issues and think what am I going to do when I'm in that sort of situation” (S2). The digital reflections created a safe place to disagree with ideas: “because I could then go vent ... when I can’t agree to the interpretation of that particular author with that particular issue. Then I was better able to articulate that in the assignments” (S6). It was also a way of monitoring their own progress in learning over time: “I have enjoyed writing these reflections each week. As now as I Australasian Journal of Educational Technology, 2021, 37(3). 30 am writing the last one for the assignment, I can see how I have travelled across this semester” (S8). The digital journal was an individual dialogic space, as it was working like a digital listener or thinking partner. When asked about the value of the collective feedback loop discussions in tutorials, the students did not place as much value on them as the staff did: I can only remember a few of those feedback loops. I remembered some of them were particularly about the time pressures that people were under and very strongly I related to that, and it was really nice to know that other people were in the same boat. (S1) It was reassuring to see examples of reflections in the tutorials: “I don't think we really knew what to put in the reflections, and having them up in the tutorial was a bit of an exemplar type reflection” (S3). Reflections were difficult to fit into their busy lives, and even though students had the option to keep reflecting, most did not continue once the assessment deadline had passed. Students recognised the value of being required to reflect as a mandated part of assessment: While the reflections were at times difficult to complete due to work and family commitments, I did encourage me to commit to the learning and I'm glad I was made to complete them. When I say 'made', I mean that it was a pass/fail as part of the assignment. If that wasn't hanging over my shoulder, I doubt that I would've completed any of them. (S1) Faster feedback had value for students when it was deliberately linked to summative assessment and would not have had as much value for students if it had been offered as an optional support. This loop of dialogue with the research team, confirmed the pedagogic value and the enabling function of the technology. Automation was needed to sustain the value and was a topic of focus in feedback loop 4. Feedback loop 4: Dialogue between the social and the technical The fourth loop involved the use of feedback to inform sociotechnical improvements such as the automation of the feedback loop. Monthly discussions between the teachers and research team highlighted the importance of confidentiality, ease of use and what kinds of information were meaningful, which involved both sociopedagogic and technical dialogue to resolve; that is, how might both the technology and pedagogy be advanced for the purpose of improving the feedback for the students? Pedagogic concerns about the initial stability of user log in experiences in GoingOK led to technical improvements by changing the architecture of the application midway through the project. While initially this was considered a purely technical fix, we found that greater ease of students logging in to record their reflections assisted with resolving social issues like trust, evident in fewer student requests for help and more positive feedback on the experience in the institutional student surveys in Semester 2. Proposals for automation such as visual representations for student cohort data were brought to the group meetings for comment. Conducting a human-driven computational analysis process (with Jupyter Notebook) resulted in identifying group patterns in the text as well as in the reflection point data (Figure 4). Australasian Journal of Educational Technology, 2021, 37(3). 31 Figure 4. Contour map of group reflection points The point (y axis) represents “distressed” (0) to “soaring” (100) with “going ok” (50) in the centre. The week (x axis) is the week number of each reflection starting at 1 and finishing at 6. The map shows the density of the reflection points indicating where the majority of students are positioning themselves each week. The shift over time represents changes in the cohort such as a polarisation of how they are going in Week 3 and a shift to mostly positive in Weeks 4 and 5. The contour map representation of all reflection points for the group (Figure 4) proved meaningful to the lecturers, as they could see a gradual lift in positive emotions and changing patterns in the emotional intensity in the group towards the end of Week 6, which was an important pedagogic goal – that students would feel a greater sense of emotional connection to the learning. The quantitative data patterns raise questions for ongoing experimental design such as how to meaningfully associate trends in the reflection point over time with themes in the students’ writing. The computational potential for analysis was the first step towards a more fully automated system where teachers could access not only deidentified reflections but also a computational analysis of those reflections. It became clear from the lengthy sociotechnical dialogue that this feature required more development time beyond the 1-year project. Discussion: Sociotechnical insights about designing for faster feedback loops The sociotechnical approach taken with GoingOK did enable faster dialogic feedback loops to be designed and enacted within the learning context (research question 1). For teachers, the feedback about student understanding was faster as it occurred much earlier in the teaching cycle. Lecturers gained fast reassurance about the direction of teaching, made quick adaptations to their pedagogy to respond to student concerns and saved time as they were no longer having to answer as many individual student emails. The quality of student thinking in summative tasks was enhanced, and a positive culture of peer support and collaboration was built from early in the semester. Students were able to give fast feedback to themselves about how new ideas related to their professional practice early in the semester and have those impressions confirmed by lecturers and peers, so they felt on the right track (Deeley, 2018). They were also able to monitor their own progress of ideas and engage in deeper reflection, a desired feedback goal (Boud & Molloy, 2013). The enabling technology did lead to faster feedback benefits, especially in feedback loop 1 (self-reflection) and feedback loop 2 (collective dialogue). Australasian Journal of Educational Technology, 2021, 37(3). 32 With respect to the potential automation, a number of important lessons were learnt (research question 2), particularly from the third loop of reflections from teachers and students and within the fourth dialogic loop of sociotechnical dialogue. Some of the sociotechnical issues slowed down implementation of automation. Firstly, the enabling technology created social tensions as traditional concepts of feedback were challenged. Additionally, social issues slowed the technical work of automation, as the importance of trust and human meaning making were recognised. Principles resulting from the discussions in the fourth loop are discussed below to inform ongoing work in similar technical designs. Technical innovation challenged social concepts of feedback. Using GoingOK as an enabling technology where learners “make sense of information from varied sources and use it to enhance the quality of their work or learning strategies” (Carless, 2015 p. 192) challenged traditions of who usually is considered the author of feedback, and the timing, focus and purpose of feedback. More conventionally, concepts of feedback are related to specific curriculum goals realised through summative assessment tasks (Thurlings et al., 2014). When the experiences of traditional and faster feedback pedagogic approaches were analysed side by side, it was clear that the digitally mediated faster- feedback approach was different to traditional concepts of feedback in universities. Faster feedback through shared reflections was not as tightly directed towards a summative task, but about learning strategies and identities. This meant that teachers were grappling with new technical and methodological ways of working with feedback and new pedagogical and conceptual feedback concepts at the same time. The technology was a material and cultural tool that was reshaping feedback practices and conceptions (Esterhazy, 2019). Table 2 Comparing traditional and dialogic faster feedback dimensions Feedback dimension Traditional approaches to feedback Technology-enabled faster feedback through GoingOK Authors Teacher or peer as expert authors Students as authors and Lecturers as responders Activity loop Teacher --->individual student enacts ---> possibly future tasks individual student ---> Computer aggregator -- > teacher ---> group ---> enacted in ---> immediate pedagogy and learning ---> immediate and future tasks Timing After summative assessment Early and prior to summative assessment Temporal Singular Iterative, longitudinal Focus of feedback  Assessment criteria  Task performance  Not personal  Self-feedback – open and prompted  Learning performance  Personal Purposes  Improving summative performance for next task  Justifying teacher’s grades  Informing student evaluative judgement  Informing future course design  Student reflexivity  Student confidence and connection to teacher and peers  Enhanced self-regulation over time  Inform teacher’s short-term formative pedagogic adaptation Quality recognised as  Individuals mastering criteria  Closing the gap between current and desired performance  Connecting personal experience and emotions to scholarly practice  Narrating an identity of performance within the discipline, group identity and scholarly culture In future iterations, a framework such as that in Table 2 could be used to prepare teachers to think about their conceptions of feedback before they engage in a similar pedagogic innovation. Australasian Journal of Educational Technology, 2021, 37(3). 33 The pedagogic concerns of teachers were taken seriously as an opportunity for mutual learning. Sociotechnical innovation cuts across traditional epistemic boundaries, and innovation can often be slow as exploration of viewpoints involves learning across different language and conceptual foundations (Gibson, 2017). We found that committing to a principle of mutual learning through dialogic feedback in the research team made it possible, providing practical support while simultaneously working through the uncertainty that inevitably occurs in sociotechnical innovations. Automation depends on pedagogic trust. Designing for automation raised questions of trust, such as who would access what information within the feedback loop and whether the information would be secure. The students were providing personal reflections that may have influenced their relationship with the teacher or institution or potentially impacted on their grades. Teachers also needed to trust that they could manage any potential student criticism that would arise through regular GoingOK reflection opportunities. Unit coordinators (teachers in charge of the unit of study) indicated that this was particularly concerning for teachers who had previously been subject to unkind student feedback via university surveys and who experienced the “terrors of performativity” (Ball, 2003). Trust was needed to enable people to share personal reflections in a new online tool; yet trust was only generated through using the tool. As described by Kim and Peterson (2017), the “relationships between online trust and the antecedents and consequences investigated are more idiosyncratic, complex, and subtle than previously thought” (p. 51). All of these concerns generated rich discussion and sociotechnical pragmatic responses. For example, in response to the teacher’s concerns about unkind student feedback and student concerns about possible teacher retaliation, we screened comments before teachers read them, and GoingOK automatically deidentified student responses. Teachers needed to trust that they could manage issues identified through the reflections. As individual reflections in the GoingOK spreadsheet were deidentified, teachers were unable to take direct action on the basis of an individual student comment. If an individual student stated that they were distressed, teachers responded to the group as a whole to indicate general advice or to point to the help options, as the reflections were deidentified. Teachers also needed to trust that the data being collected through GoingOK would be used ethically. Australian universities are increasingly moving towards using student surveys and feedback for purposes beyond the improvement of teaching and learning, such as part of accountability and performance management systems (Blackmore, 2009). Similar concerns are being raised more broadly as digital analysis cuts across “legal, behavioural, procedural, and social boundaries”, including how to protect information privacy, and manage consent where there is a power imbalance between students, teachers and the institution (Willis et al., 2016, pp. 882–883). Two principles that guided decisions in response to these dilemmas were firstly, that students were the owners of the reflective data, and secondly, that development should be guided by what enhanced social and technical trust. This meant that the GoingOK platform needed an increased level of stability so that students could trust that it would care for their data. This informed iterative development of the software, the technical infrastructure and the way the software architecture would run on this infrastructure. These kinds of design decisions are largely invisible to end users; however, we found that they can have a significant impact on social enablers like trust. Pedagogic resources such as videos and tip sheets for students were developed during the project to promote familiarity and trust in the tool. The GoingOK user information was also developed further by two of us (Gibson and Willis), and was made available on the support website. The pragmatic intent of benefit for users informed the development of social and technical innovations. RWA innovations required human insights and an initial human workload increase. The technical potential for faster feedback was demonstrated through making feedback available to students and teachers sooner than post-summative feedback and in highlighting how even early steps towards the automation of analytics (Figure 4) can support teachers to find latent patterns in the feedback data. Automation, however, depended on the initial manual analysis of reflections to decide what patterns were meaningful. Automation also needed to attend to stakeholder concerns. As these concerns emerged in real time throughout the project, design of the software to perform computational analysis occurred asynchronously after the concerns were raised. Also, as the pedagogical structure of feedback loops drove the project, the technology needed to be ready for use from the beginning of a teaching period leaving very Australasian Journal of Educational Technology, 2021, 37(3). 34 small windows of time for complex software development tasks prior to the project start. The desired interplay between the pedagogical and the technical required a level of concurrency. However, pausing the pedagogical while undertaking the technological was not an option for this kind of work. The initial computational analysis did reveal the potential for automation, but also the inevitability of asynchronous design, and initial human workload intensification, when balancing pedagogic and technical priorities. Technical responses to pedagogical questions can range from the trivial to the impossible and rarely can be determined in advance. Care needs to be taken that technical development is determined by value to the learning, rather than by what is easy to implement within time constraints. Technological changes that have most value for learning may also incur the largest cost of change. For example, technological change may be undesirable where pedagogic consistency of experience is required. These interactions between the pedagogical and technological loops need to be balanced wisely with decisions being made at the time of the interaction between the loops. The team’s commitment to collective dialogue and mutual learning was a key principle that enabled progress and could be applied in other contexts. Implications and limitations This research has implications for future work in using technology with feedback. The research also exposed some limitations in our approach. The sociotechnical approach guided by a pragmatic inquiry style method provided a clear focus on the practical effects of faster feedback for students. However, in doing so, it moves outside of educational and technical disciplinary research conventions. For example, the approach is unlikely to meet the expectations of pure social research as it is driven by the anticipated practical effects rather than by social theory. Similarly, it is unlikely to meet the expectations of purely technical research as it is not driven by analytical data and technical benchmarking. However, importantly, the sociotechnical approach allows for a principled integration of both pedagogy and technology, which is not a feature in either domain’s dominant research paradigms. The implication is that accepting these limitations is a necessary part of engaging in this type of sociotechnical transdisciplinary work. It also has implications for the need for transdisciplinary research teams, where members can engage in ongoing dialogue. Practical benefits for students in the dialogic process could not have happened without the sociotechnical approach. Students benefited from feedback with a fundamentally different nature than the status quo, and the dialogic qualities of that feedback clearly resulted from the interaction between the social and the technical. Creating similar feedback without the technology would have been very difficult if not impossible. Similarly, a purely computational approach is highly unlikely to have been successful. The dialogic feedback loops involved both pedagogy and technology, and while interconnected, they were also asynchronous. The lack of synchronicity was a limitation in achieving the aim of automated RWA and is related to the time limitation of the project. Future project designs need to accommodate the time it takes for technical development while somehow maintaining the integrity of pedagogical feedback loops without disruption. This suggests a longer project time and an agile style of project development where pedagogically critical tasks are undertaken in a responsive manner. Such an approach may not necessarily be efficient from a software development point of view but could potentially yield greater technological and pedagogical value over time. Finally, the project provided insight into the complexity of researching and working at the intersection of pedagogy in action, software development and learning analytics. Deeply reflexive discussions within the research team enabled local responses, but also principles that can apply to other contexts. Further development of the complex software that will eventually automate the analysis of the data occurred in an asynchronous time cycle. The manual data analysis that informed the feedback loops was a cognitively demanding and time-intensive task involving two team members analysing the 500 or so deidentified reflections submitted per week. Although this was a labour-intensive component of the study and needed to be undertaken each week of the teaching semesters, it has generated vital information that will inform the design of potential software for automating feedback in the future. Australasian Journal of Educational Technology, 2021, 37(3). 35 Conclusion This study adds to the growing field researching the potential pedagogical applications of digital tools and learning analytics to enhance quality learning and teaching. Significantly, this study identifies four distinct types of digitally mediated feedback loops. Establishing the distinct characteristics of each feedback loop and the theoretical frameworks that inform these complex learning and teaching processes provides an evidence base for conceptualising and using feedback in a more dialogical way. The series of feedback loops generated regular opportunities for dialogue and problem-solving, normalised the challenges of study and enhanced a sense of social connection across the cohort. These were positive impacts for both students and teaching staff. In theorising sociotechnical approaches to faster feedback in higher education, this study provides insight into the potential role that collaboration between educators and digital developers can play in creating meaningful and productive learning and teaching experiences during the teaching cycle, while exploring future possibilities for automation. Acknowledgements The research team would like to acknowledge the teachers (lecturers and tutors) and students who shared their learning, and the University Learning and Teaching Committee for investing in innovation. References Archer, M. S. (2007). Making our way through the world: Human reflexivity and social mobility. Cambridge University Press. https://doi.org/10.1017/CBO9780511618932 Ball, S. J. (2003). The teacher's soul and the terrors of performativity. Journal of Education Policy, 18(2), 215–228. https://doi.org/10.1080/0268093022000043065 Beaumont, C., O’Doherty M., & Shannon, L. (2008). Staff and student perceptions of feedback quality in the context of widening participation. The Higher Education Academy. https://www.advance- he.ac.uk/knowledge-hub/beaumont-c-odoherty-m-and-shannon-l-staff-and-student-perceptions- feedback-quality Bernaitis, N., Baumann-Birkbeck, L., Alcorn, S., Powell, M., Arora, D., & Anoopkumar-Dukie, S. (2018). Simulated patient cases using DecisionSimTM improves student performance and satisfaction in pharmacotherapeutics education. Currents in Pharmacy Teaching & Learning, 10(6), 730–735. https://doi.org/10.1016/j.cptl.2018.03.020 Blackmore, J. (2009). Academic pedagogies, quality logics and performative universities: Evaluating teaching and what students want. Studies in Higher Education, 34(8), 857–872. https://doi.org/10.1080/03075070902898664 Boud, D., & Molloy, E. (2013) Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712. https://doi.org/10.1080/02602938.2012.691462 Brady, M., Devitt, A., & Kiersey, R. A. (2019). Academic staff perspectives on technology for assessment (TfA) in higher education: A systematic literature review. British Journal of Educational Technology, 50(6), 3080–3098. https://doi.org/10.1111/bjet.12742 Carless, D. (2015). Excellence in university assessment: Learning from award-winning practice. Routledge. https://doi.org/10.4324/9781315740621 Carless, D. (2019). Feedback loops and the longer-term: Towards feedback spirals. Assessment & Evaluation in Higher Education, 44(5), 705–714. https://doi.org/10.1080/02602938.2018.1531108 Deeley, S. (2018). Using technology to facilitate effective assessment for learning and feedback in higher education. Assessment & Evaluation in Higher Education, 43(3), 439–448. https://doi.org/10.1080/02602938.2017.1356906 Esterhazy, R. (2019). Re-conceptualizing feedback through a sociocultural lens. In M. Henderson, R. Ajjawi, & D. Boud (Eds.), The impact of feedback in higher education (pp. 67–82). Palgrave Macmillan https://www.doi.org/10.1007/978-3-030-25112-3 Gibson, A. (2017). Reflective writing analytics and transepistemic abduction [Doctoral dissertation, Queensland University of Technology]. ePrints. https://doi.org/10.5204/thesis.eprints.106952 https://doi.org/10.1017/CBO9780511618932 https://doi.org/10.1080/0268093022000043065 https://www.advance-he.ac.uk/knowledge-hub/beaumont-c-odoherty-m-and-shannon-l-staff-and-student-perceptions-feedback-quality https://www.advance-he.ac.uk/knowledge-hub/beaumont-c-odoherty-m-and-shannon-l-staff-and-student-perceptions-feedback-quality https://www.advance-he.ac.uk/knowledge-hub/beaumont-c-odoherty-m-and-shannon-l-staff-and-student-perceptions-feedback-quality https://doi.org/10.1016/j.cptl.2018.03.020 https://doi.org/10.1080/03075070902898664 https://doi.org/10.1080/02602938.2012.691462 https://doi.org/10.1111/bjet.12742 https://doi.org/10.4324/9781315740621 https://doi.org/10.1080/02602938.2018.1531108 https://doi.org/10.1080/02602938.2017.1356906 https://doi.org/10.5204/thesis.eprints.106952 Australasian Journal of Educational Technology, 2021, 37(3). 36 Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017, March). Reflective writing analytics for actionable feedback. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 153–162). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027436 Gibson, A., & Lang, C. (2018). The pragmatic maxim as learning analytics research method. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 461– 465). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170384 Gibson, A., & Lang, C. (2019). Quality indicators through learning analytics. In M. Peters (Ed.), Encyclopedia of teacher education (pp. 1–6). Springer. https://doi.org/10.1007/978-981-13-1179- 6_203-1 Gravett, S., & Petersen, N. (2002). Structuring dialogue with students via learning tasks. Innovative Higher Education, 26(4), 281–291. https://doi.org/10.1023/A:1015833114292 Han, J., & Finkelstein, A. (2013). Understanding the effects of professors’ pedagogical development with clicker assessment and feedback technologies and the impact on students’ engagement and learning in higher education. Computers & Education, 65, 64–76. https://doi.org/10.1016/j.compedu.2013.02.002 Henderson, M., Boud, D., & Molloy, E., Dawson, P., Phillips, M., Ryan, T., & Mahoney, P. (2018). Feedback for learning: Closing the assessment loop (Final report). Australian Government. https://ltr.edu.au/resources/ID16-5366_Henderson_Report_2018.pdf Henderson, M., Selwyn, N., & Aston, R. (2017). What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning. Studies in Higher Education, 42(8), 1567– 1579. https://doi.org/10.1080/03075079.2015.1007946 Hill, J., & West, H. (2020). Improving the student learning experience through dialogic feed-forward assessment. Assessment & Evaluation in Higher Education, 45(1), 82–97. https://doi.org/10.1080/02602938.2019.1608908 Holbrey, C. (2020). Kahoot! Using a game-based approach to blended learning to support effective learning environments and student engagement in traditional lecture theatres. Technology, Pedagogy and Education, 29(2), 191–202. https://doi.org/10.1080/1475939X.2020.1737568 Nash, R., & Winstone, N. (2017). Responsibility-sharing in the giving and receiving of assessment feedback. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01519 Orsmond, P., Merry, S., & Reiling, K. (2005). Biology students' utilization of tutors' formative feedback: A qualitative interview study. Assessment & Evaluation in Higher Education, 30(4), 369–386. https://doi.org/10.1080/02602930500099177 Pennebaker, J., & Seagal, J. (1999). Forming a story: The health benefits of narrative. Journal of Clinical Psychology, 55(10), 1243–1254. https://doi.org/10.1002/(SICI)1097-4679(199910)55:10<1243::AID- JCLP6>3.0.CO;2-N Ryan, M. (Ed.). (2015). Teaching reflective learning in higher education: A systematic approach using pedagogic patterns. Springer Nature. https://doi.org/10.1007/978-3-319-09271-3 Shibani, A., Knight, S., Buckingham Shum, S., & Ryan, P. (2017, January). Design and implementation of a pedagogic intervention using writing analytics. In W. Chen et al. (Eds.), Proceedings of the 25th International Conference on Computers in (pp. 306–315). Asia-Pacific Society for Computers in Education. https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/sites/default/files/proceedings/mai n/C3/Design%20and%20Implementation%20of%20a%20Pedagogic%20Intervention%20Using%20 Writing%20Analytics.pdf Thurlings, M., Mermeulen, M., Bastiaens, T., & Stijnen, S. (2014). The role of feedback and social presence in an online peer coaching program for student teachers. Australasian Journal of Educational Technology, 30(3), 326–341. https://doi.org/10.14742/ajet.636 Wickson, F., Carew, A. L., & Russell, A. W. (2006). Transdisciplinary research: Characteristics, quandaries and quality. Futures, 38(9), 1046–1059. https://doi.org/10.1016/j.futures.2006.02.011 Willis, J., Crosswell, L., Morrison, C., Gibson, A., & Ryan, M. (2017). Looking for leadership: The potential of dialogic reflexivity with rural early-career teachers. Teachers and Teaching, 23(7), 794– 809. https://doi.org/10.1080/13540602.2017.1287695 Willis, J., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: a typology derived from a cross-continental, cross-institutional perspective. Educational Technology Research and Development, 64(5), 881–901. https://doi.org/10.1007/s11423-016-9463-4 Winstone, N., & Carless, D. (2020). Designing effective feedback processes in higher education: A learning-focused approach. Routledge. https://doi.org/10.1145/3027385.3027436 https://doi.org/10.1145/3170358.3170384 https://doi.org/10.1007/978-981-13-1179-6_203-1 https://doi.org/10.1007/978-981-13-1179-6_203-1 https://doi.org/10.1023/A:1015833114292 https://doi.org/10.1016/j.compedu.2013.02.002 https://ltr.edu.au/resources/ID16-5366_Henderson_Report_2018.pdf https://doi.org/10.1080/03075079.2015.1007946 https://doi.org/10.1080/02602938.2019.1608908 https://doi.org/10.1080/1475939X.2020.1737568 https://doi.org/10.3389/fpsyg.2017.01519 https://doi.org/10.1080/02602930500099177 https://doi.org/10.1002/(SICI)1097-4679(199910)55:10%3C1243::AID-JCLP6%3E3.0.CO;2-N https://doi.org/10.1002/(SICI)1097-4679(199910)55:10%3C1243::AID-JCLP6%3E3.0.CO;2-N https://doi.org/10.1007/978-3-319-09271-3 https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/sites/default/files/proceedings/main/C3/Design%20and%20Implementation%20of%20a%20Pedagogic%20Intervention%20Using%20Writing%20Analytics.pdf https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/sites/default/files/proceedings/main/C3/Design%20and%20Implementation%20of%20a%20Pedagogic%20Intervention%20Using%20Writing%20Analytics.pdf https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/sites/default/files/proceedings/main/C3/Design%20and%20Implementation%20of%20a%20Pedagogic%20Intervention%20Using%20Writing%20Analytics.pdf https://doi.org/10.14742/ajet.636 https://doi.org/10.1016/j.futures.2006.02.011 https://doi.org/10.1080/13540602.2017.1287695 https://doi.org/10.1007/s11423-016-9463-4 Australasian Journal of Educational Technology, 2021, 37(3). 37 Yang, M., & Carless, D. (2013). The feedback triangle and the enhancement of dialogic feedback processes. Teaching in Higher Education, 18(3), 285–297. https://doi.org/10.1080/13562517.2012.719154 Corresponding author: Jill Willis, jill.willis@qut.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Willis, J., Gibson, A., Kelly, N., Spina, N., Azordegan, J. M., & Crosswell, L. (2021). Towards faster feedback in higher education through digitally mediated dialogic loops. Australasian Journal of Educational Technology, 37(3), 22–37. https://doi.org/10.14742/ajet.5977 https://doi.org/10.1080/13562517.2012.719154 mailto:jill.willis@qut.edu.au https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5977