Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 1 Thematic analysis of individual feedback: Improving cohort feedforward Richard Edward Meredith University of Greenwich Abstract Assessment literacy is important for students’ academic success. This multi-case study explores scholarly writing mistakes commonly made by direct-entrant international students at level 7. The data consists of assessment data from 150 student scripts. Results show common themes related to cross-cultural differences for international students who began their masters-level higher educational experience in a new culture and new environment. It contributes common factors, hitherto hidden in student assessment data in Turnitin’s global writing technology. Keywords: Assessment and marking; thematic analysis; assessment literacy; feedforward, business education 1. Introduction Turnitin assessment text data can be made visible rather than remaining unseen, unnoticed and therefore unactionable (Bienkowski et al., 2012). A poster presentation at the University of Greenwich Learning and Teaching Festival 2019 became a transformative learning experience that led to the perception of Turnitin assessment data as a new data source for assessment analysis, modelled using activity theory. 2. Background The poster reported an inductive study which developed an in-depth description and analysis of 150 cases of level 7 student essay assessments. The Managing Across Cultures module was taught to 389 students in a United Kingdom (UK) MBA/MA international Business degree programme for new entrant international students, mainly from the continent of Asia. The learning aims concerned the topic of cross-cultural management and were scheduled for teaching in the first semester of arrival in the UK, alongside a module about foundations of scholarship. The poster presented at the Learning and Teaching Festival aimed to provide a case study of how one sessional worker adopted qualitative data analysis (QDA) software to manage the complexity and time pressure of high-volume assessment and marking. The Turnitin software is designed to facilitate a single set of marks and feedback per student script, but doesn't currently include any features designed specifically to support either document management of multiple marking teams or assessment analysis across multiple students. In the case study, 150 student scripts were coded in the same way as in Turnitin. The main rationale for doing so was to consult – more easily – those written and oral briefings, assessment criteria, module handbooks and additional study guides, descriptors and Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 2 samples of marked work as add document complexity to the enterprise of marking. Accomplishing this activity smoothly – often in ten rather than fifteen days, irrespective of the assessment workload incurred by any one individual – is no mean feat. The assessment criteria defined in the module handbook were based on a portfolio assignment of three tasks: a) cultural interview and personal profile 550 words (15%); b) reflection on cross-cultural competencies 450 words (15%); and c) case analysis using cross-cultural management theories, 2000 words (70%). 3. Research method The study aimed to provide a description of common areas of assessment feedback on the learning of a large cohort of students. Such an analysis is in line with concerns of educational practitioners that “feedback is a troublesome issue in higher education” (Nicol et al., 2014, p.102). The situational influence on the students’ learning gain was that these students were mostly from non-western higher education (HE) systems and were entering level 7 as direct- entry international students studying Cross-Cultural Management. A case study protocol and multiple data sources were used to provide data triangulation and to enhance the reliability and validity of findings (Denzin 1989; Yin 2015). Thematic analysis of data within each case and analytic comparison of data across cases were conducted to unveil, along key themes, similarities and differences in the evidence (Neuman 2006). At the same time, relevant module handbook documents and teaching team guidance and communications materials were analysed for corroborative or contradictory evidence to enhance the validity of the case study findings (Yin, 2015). 4. Results As it was exploratory in nature, an iterative approach to analysis and identification of themes and activities was employed in the multiple-case study research. By means of the software programme mentioned above – to code and refine the rich data through an immersion/crystallisation process enabling back-and-forth working between the themes and the database (Cresswell and Poth, 2018) – three themes were identified in the feedback: 4.1 Theme 1: scholarly argument missing one or more of the following components: a) proposition; b) reasoning – usually from theory and supported by evidence; c) conclusion – the solution to the proposition and justified by the reasoning. 4.2 Theme 2: lack of resourcefulness in data collection and of criticism in data selection; weakness in acknowleging sources. 4.3 Theme 3: discomfort with writing reflectively – a common assessment type across the programme. 5. Discussion The results indicated a potential correlation between student nationality and weak foundation in scholarship for demonstrating learning aims of the module in Cross-Cultural Management. The module taught students about the need to understand cultures and how they can differ – sometimes significantly – between nations and regions. However, in the programme’s scheduling, this aspect was built neither into socialisation nor into creation and reinforcement of new normative beliefs. The results point towards the need for research on the impact of culture on assessment and, especially, a) the influence of group interests within Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 3 collective cultures (Hofstede, 1983) and b) the respective influence of neutral and affective cultures (Trompenaars and Hampden-Turner, 2012) on normative beliefs about argumentative and reflective writing. Transformative learning The by-product of assessment and marking outside the Turnitin system was that otherwise tacit knowledge of common assessment errors across a cohort of students was now captured for inductive thematic analysis and results were available for a) summary feedback when marks are released; b) narrative data for annual module reporting; c) learning gain data for reflecting on future assessment and module design. Several conference delegates identified Turnitin Assessment analysis as an assessment analytic. My frame of reference to produce the conference poster was as a sessional worker coping with the process of marking. Looking back, these delegate comments were integral to self-reflection and learning, helping to shape my thinking that thematic analysis of assessment data on any module could conceivably be used for re-designing the learning environment for the following academic year. Mezirow (2000) suggested that transformative learning only occurs when problematic frames of reference that fix assumptions and expectations are consciously unlearned. “A frame of reference encompasses cognitive, conative, and emotional components, and is composed of two dimensions: habits of mind and a point of view.” (Mezirow (1997, p.5). I had read about Mezirow’s transformative learning in a recent book – ‘Contemporary Theories of Learning’ (Illeris, 2018) – and the delegates’ comments stimulated the metacognitive process of reassessing reasons. Mezirow (2003) considered that an open mind and the ability to listen empathetically to others are necessary for reflective practice to occur. Potential conceptual framework from transformative learning Having been unfrozen (Lewin, 1943) from my frame of reference of a marking process, what new frame pertaining to student learning activities might I change to? Activity theory (Engestrom, 1987) systemically examines the context in which learning occurs as well as the design process. The activity ‘system’ conceived by Engestrom appears below (figure 1). The top half (shown as production), dynamically links the subject who performs an activity to the object of the activity and the tools that the subject uses in the activity. Below this triangle is the context – the division of labour associated with the activity within the community or organisation, members of which share a set of social meanings or rules for conducting such activities. Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 4 Figure 1: Activity theory model (Source: Engestrom, 1987) Jonassen and Roner-Murphy (1999, p.62) argued that activity theory (Engestrom, 1987) provides a powerful conceptual framework for designing a learning environment because “it posits that conscious learning emerges from activity (performance), not a precursor to it”. Xing et al., (2014) operationalised activity theory in a computer-supported collaborative learning computer environment to develop a student performance prediction model based on the six activity theory variables – subject, object, tools, division of labour, rules and community. Jonassen and Roner-Murph relied on very different epistemic assumptions about the design of a student learning environment from traditional methods which assume relevant knowledge to be embedded in the instruction for transfer to the learner in any context. They explicate, using activity theory, the methods for creating a constructive learning environment. The interpersonal dialogue at the conference about my poster led me to a new frame of reference. How about a theoretically grounded factorisation of three sets of data in order both to improve module design and adapt tutoring to individual student contexts? Namely, the integration of – as an activiy theory tool - of: a) historical assessment data, as a community factor; b) student categorical data, as a subject factor; and c) learning analytic data about in- class (audience participation tools) and on-line learning activities (virtual learning environment analytics). Figure 2 models the structure of such an activity system, extending Engestrom (1999) through inclusion of assessment data in the community component. Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 5 Figure 2 Profiling Turnitin data – an adaptation of activity theory model. Source: Author, derived from Engestrom (1987) A description of activity theory operationalisation in order to make sense of historical Turnitin assessment data as a learning catalytic is set out in Table 1. Table 1: Student performance prediction model operationalisation of activity theory Dimension Definition Production Learning involves a subject student; the mental object of activity, being learning; the learning resource tools such as the VLE that are used in the activity. As activity systems are conceived to be socially and contextually bound, the actions and operations that affect an outcome include the rules, community and division of labour. Subject Individual students who engage in the activity to achieve the object of learning. Object Completing learning tasks – represents the intention that motivates the activity. Tools Computers, online tools, systems, and environments that mediate the learning activity. Division of labour Individual assignments within the overall activity, which is also mediated by rules and social negotiation. Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 6 Dimension Definition Rules Implicit and explicit rules and guidelines that constrain the activity. For example, institutional academic rules of student behaviour and quality standards and specific rules set by module leaders for learning tasks (explicit). An individual student can use only the function residing in the supporting tools (implicit). Community The community of students at the same academic level who have previously completed the activity of learning. The customary areas of difficulty or errors in completion form the context of the activity in which it operates. The activity theory factor called ‘community’ in figure 2 and table 1 is perhaps contextually the most relevant to the design of a module. Jonassen and Roner-Murphy (1999) argued that traditional methods of task analysis focused only on the technical core of performance, ignoring the contexts within which learning occurs. Historical assessment text that identifies common themes and associated student categories offers the potential to yield a new source of rich context that is important when designing instruction. Conclusion Interest has increased in analytics as part of the solution to many issues in HE (Baker and Yacef, 2009; Romero and Ventura, 2010). However, a practical means of identifying academic at-risk students before the start of term appears to have eluded researchers so far. Mezirow (2000) believed that educational interventions are necessary to ensure that the learner acquires the understandings, skills and dispositions essential for transformative learning. This article offers a model for Turnitin assessment text as a assessment analytic based on transformative learning experiences at the University of Greenwich Learning and Teaching Festival 2019. I hope this reflective report demonstrates that the festival was an effective intervention. Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 7 Reference list Baker, R.S. and Yacef, K. (2009) ‘The state of educational data mining in 2009: A review and future visions.’ JEDM| Journal of Educational Data Mining, 1, 3-17. Available at: https://doi.org/10.5281/zenodo.3554657 (Accessed: 24 September 2019). Bienkowski, M., Feng, M. and Means, B. (2012) ‘Enhancing teaching and learning through educational data mining and learning analytics: An issue brief.’ US Department of Education, Office of Educational Technology, 1-57. Available at: https://mobile.eduq.info/xmlui/handle/11515/35829 (Accessed: 24 September 2019). Creswell, J.W. and Poth, C.N. (2017) Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, California: Sage Publications. ISBN 978-1-5063- 3019-8. Denzin, N.K. (1989) Interpretive interactionism. Applied social research methods series. Newbury Park, CA: Sage Publications. ISBN 9780761915140. Engestrom, Y. (1999) 23 Innovative learning in work teams: Analyzing cycles of knowledge creation in practice. Perspectives on activity theory. Cambridge: Cambridge University Press. ISBN 0-521-43730-X. Hofstede, G. (1983) ‘The cultural relativity of organizational practices and theories.’ Journal of International Business Studies 14, 75. Available at: https://doi.org/10.1057/palgrave.jibs.8490867 (Accessed: 2 September 2019). Illeris, K. (ed.) (2018) Contemporary theories of learning: learning theorists... in their own words. (Second edition.) Milton Park, Abingdon, Oxon: Routledge. ISBN 978-1-138-55049-0. Jonassen, D.H. and Rohrer-Murphy, L. (1999) ‘Activity theory as a framework for designing constructivist learning environments.’ Educational Technology Research and Development 47, 61-79. Available at: https://doi.org/10.1007/BF02299477 (Accessed: 25 September 2019). Lewin, K. (1943) ‘Defining the field at a given time.’ Psychological review. Available at: https://doi.org/10.1037/h0062738 (Accessed: 15 January 2016). Mezirow, J. (2003) ‘Transformative learning as discourse.’ Journal of transformative education 1, 58-63. Available at: https://doi.org/10.1177/1541344603252172 (Accessed: 28 September 2019). Mezirow, J. (2000) Learning as Transformation: Critical Perspectives on a Theory in Progress. The Jossey-Bass Higher and Adult Education Series. ERIC. San Francisco, CA: Jossey-Bass ISBN ISBN-0-7879-4845-4. Mezirow, J. (1997), Transformative Learning: Theory to Practice. New Directions for Adult and Continuing Education, 1997: 5-12. doi:10.1002/ace.7401 (Accessed: 28 September 2019). https://doi.org/10.1007/BF02299477 Conference Reflections Compass: Journal of Learning and Teaching, Vol 13, No 1, 2020 8 Neuman, W.L. (2006.) Analysis of qualitative data in Social research methods: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage, 457-489. Nicol, D., Thomson, A. and Breslin, C. (2014) ‘Rethinking feedback practices in higher education: a peer review perspective.’ Assessment & Evaluation in Higher Education, 39, 102-122. Available at: https://doi.org/10.1080/02602938.2013.795518 (Accessed: 12 November 2019). Romero, C. and Ventura, S. (2010) ‘Educational data mining: a review of the state of the art.’ IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 40, 601-618. Available at: https://doi.org/10.1109/TSMCC.2010.2053532 (Accessed: 24 September 2019). Trompenaars, A., Hampden-Turner, C. (2012) Riding the waves of culture : understanding diversity in global business. (Third edition) New York: McGraw-Hill. ISBN 0786311258. Xing, W., Guo, R., Petakovic, E. and Goggins, S. (2015) ‘Participation-based student final performance prediction model through interpretable Genetic Programming: Integrating learning analytics, educational data mining and theory.’ Computers in Human Behavior, 47, 168-181. Available at: https://doi.org/10.1016/j.chb.2014.09.034 (Accessed: 24 September 2019). Yin, R.K. (2015) Qualitative research from start to finish. New York: Guilford Publications. ISBN 978-1-4625-1797-8. https://doi.org/10.1016/j.chb.2014.09.034