Irish Journal of Technology Enhanced Learning “How are you getting on?” Evaluating discussion boards as assessment for social care work-based placement * Karolyn McDonnell *1 1South East Technological University Abstract This research evaluates the use of asynchronous online discussion boards for assessing work- based placement in a social care degree programme. Owing to a scarcity of educational technologies in the programme at the time (pre-COVID), discussion boards were introduced to assess students' reflections on placement experiences. One objective was to facilitate knowledge transfer: sharing their experiences of placement might prompt a stronger understanding of applying professional competencies required for the role. Another was to enable peer-learning, reflection and peer feedback to take place. . Staff (instructors) and students’ attitudes were explored to identify the acceptability of using discussion boards for work-based placement. The components of discussion boards for this purpose are identified, and the considerations to be made when implementing discussion boards to support the development of reflection skills are presented. Using a mixed methods sequential complementarity research design, two student focus groups were held following the distribution of an online survey. Four instructors were interviewed about their attitudes and experience of using the boards for the first time. Finally, a code adapted from Henri’s (1992) computer mediated conferencing framework and Kolb’s (1984) learning cycle was created to analyse the types of reflectivity and depth of interactivity within the group discussions. Whilst discussion boards are a useful platform to facilitate student to student conversation, their use in this instance fell short of maximising their potential to achieve the learning objectives and to foster substantive and meaningful learning. Students found the interface cumbersome and unappealing and there was a lack of consistency with instructor’s provision of feedback where one instructor admitted providing none. Despite this, four components of discussion boards were identified and key considerations when using discussion boards to encourage reflective activity in social care work-based placement are proposed and discussed. 1. Introduction Students must demonstrate proficiency in eighty listed threshold standards to be deemed competent to enter the professional social care register (CORU, 2017). Digital literacy skills, communication technologies (CORU, 2017, p.6), and the ability to evaluate and reflect critically on one’s own professional practice (CORU, 2017, p.8) are included in the listed standards. The data reported here is part of a broader study. A summation of some literature focusing on communication, engagement and reflection using discussion boards is presented, the methodology is described, and some of the most salient findings from the evaluation are discussed in efforts to support future iterations of this assessment in a similar context, with recommendations emerging for instructors and students. 2 1.1 Communication and Constructivism Social care scholarship draws from the socio-constructivist theory of learning posited by Vygotsky (1978) who stressed the fundamental role of interaction to cognition and learning development. Social interaction is vital to the learning process providing ‘a means for students to view topics from multiple perspectives and enhance their critical thinking and problem-solving skills’ (Hurst et al., 2013, p.390). Bates (2005) proffered a distinction between participatory activities in the online environment by categorising the social component as activities between two or more stakeholders, and the cognitive component as between the learner and the task activity. Participation in an online domain can be a perfunctory activity. Logging in and providing intermittent, cursory commentary for example merely confirms the learner’s attendance and suggests attention to the task or activity at hand: the quality of the learner’s engagement remains ambiguous (Douglas et al., 2020). The moderator’s key function is to oversee and facilitate communication in online discussions. Their behaviour and interaction with discussants have the biggest impact on whether the discussion achieves what it sets out to achieve (Donnelly and Gardner, 2011; Loncar et al., 2014). Loncar et al., (2014) state that ‘an un-mediated, uncontrolled, or un- facilitated discussion will likely not result in an ‘effective’ discussion, learning, or knowledge construction’ (Loncar et al., 2014, p.98). Moderator absence has a profound influence on students’ engagement with and perceptions of the usefulness of the discussion. Moderation is crucial to keep topics and engagement focused as well as helping students acclimate to the online environment and participate productively. Indeed Salmon (2003) describes the moderator as a social host, where Xie et al., (2018) suggest they serve a leadership role, emphasising its significance within social constructivism theory. 1.2 Factors of Engagement Guidance and technical support are essential to foster an integrative online community. The actions of the moderator inextricably influence the students’ engagement with and experience within the discussion. Instructors often struggle with their moderator role to promote and develop a lively and productive online discussion (de Lima et al., 2019). Salmon (2014) devised a five-stage model highlighting the moderators’ tasks in online discussions. Ease of accessibility to a device, the platform and systems and Wi-Fi increases users’ motivation to engage online. Following this, effective moderation involves providing induction, prompting social exchanges, facilitating the discussions, and querying and supporting users’ conclusions (Salmon, 2014). Grade provision also affects how students engage and participate in online discussions. Ransdell et al., (2018) warn that ungraded boards will not attract any appeal for learners. In cognate health related disciplines, polarising results emerge from studies exploring discussion boards as an augmented learning activity. Whilst Caldarola (2014) and Giacumo et al., (2013) concluded that there was no improvement in student’s final grades correlating to their Irish Journal of Technology Enhanced Learning 3 discussion board activities, Taradi and Taradi (2004) found their students earned higher mean grades, suggesting that substantive learning from discussion boards is entirely subjective and influenced by a myriad of factors. To encourage engagement with feedback, Kuepper-Tetzel and Gardner (2021) propose withholding grade: students are more likely to re-read their submissions, engage with feedback and recognise their own strengths and areas for development if grade is not their focus. Kohn (1994) questioned the reciprocal enterprise of graded assessment, impelling a student to perform based on their desire for validation, rather than fostering a learning mind- set. They argue that timely and actionable feedback is inherently more useful to encourage students’ acquisition of self-determination, motivation and confidence, supporting learners to reach their full academic potential (Wormeli, 2006; Deci et al., 2011). 1.3 Reflection Learning to reflect on professional practice is learning how to learn from practice experiences. Social care students must demonstrate their ability to evaluate their development as a professional using critical reflection on real-world experiences, encompassing their positionality, life experiences, biases and personal values (CORU, 2017). Assessing reflection continues to be a challenging undertaking in higher education. A key argument led by Ixer (1999), Russell (2013) and Yip (2005), has produced contrasting discourse centring on the indeterminate power differential that can exist between staff and students. Ixer suggests as a practice, its integrity is compromised because of this power imbalance, stating the ‘harm’ of assessing ‘vulnerable learners who do not happen to fit into the assessors' own ideas of what they believe reflective learning to be’ (Ixer, 1999, p.514). Staff with poorly formed conceptions of reflection are ill equipped to assess what constitutes ‘good’ reflection. A sentiment shared amongst a select few academics, Russell (2013) advances the argument that those who do not engage in systematic reflective practice are not well placed to teach and assess it. Yip (2005) emphasises the self-involved nature of reflection and it is potential to trigger uncomfortable feelings and deep-seated emotions, warning against the ‘highly destructive’ practice of exploring these emotions under inappropriate conditions - in an oppressive environment, for example, or to a highly critical supervisor. (Yip, 2005, p.785) Similarly, students bearing unresolved trauma or a negative self-image may internalise critical comments from an unsympathetic or untrained assessor. Sumsion and Fleet (1996), similarly documented the limited availability of an impartial instrument to measure or assess reflection in learners. In 2000 however, Kember et al., devised a four-scale measurement which has been deemed robust and rigorous within the academic community and has been reliably used in a number of studies, Lucas et al., (2006), Tsingos et al., (2015) and Perkowska-Klejman and Odrowaz-Coates, (2019) for example. This measurement might well reduce ambiguity for instructors and address the arguments mentioned by asking respondents to self- assess their development of reflective thinking under four domains: habitual action, recognition, reflection and critical reflection (Kember et al., 2000). 4 2. Methodology A combination of qualitative and quantitative methods in a sequential complementarity approach was used for this investigation. The data comprised of the results of an online survey (n=34), and feedback from two social care student focus groups (n=6 each). The survey was devised from Kember et al’s., (2000) Questionnaire for Reflective Thinking. Four instructor interviews and the focus group transcripts were thematised and categorised using Braun and Clarkes (2012) thematic analysis framework. Finally, a code adapted from Henri’s (1992) content analysis framework and Kolb’s (1984) learning cycle was applied to a group transcript. This code was used to analyse the semantic content of discussions to ascertain the extent to which reflection was evident in the students’ accounts of experiences, as well as identifying the extent to which the students were interacting with and learning from each other, in contrast to providing a series of monologues. According to Kolb, the four stages of learning are Concrete Experience (describing the experience, etc.,), Reflective Observation (reviewing thoughts, emotions, feelings, etc.,), Abstract Conceptualisation (extracting deeper insight, creating links between the experience and new learning, etc.,) and Active Experimentation (realisation of learning, resolution towards action, etc.,) (Kolb, 1984). Henri (1992) developed a framework of five dimensions of interaction within online communications: Participative, Social, Interactive, Cognitive and Metacognitive with specific indicators for each to mitigate dissention from multiple reviewers. Both Kolb and Henri’s codes were applied in tandem to identify whether students were achieving the markers of reflection (according to Kolb’s stages) and what types of, interactions were taking place within the discussion (according to Henri’s framework). Limitations Data collection was delayed to the academic year following placement instead of immediately after placement due to timing issues with ethical approval. Following ethical approval from the Institute of Technology, Carlow ethics committee, consent was acquired for just one group transcript to be analysed rendering this aspect of the investigation exploratory and its findings not transferable. Final stage data analysis would have benefitted from inter-rater reliability and as a result, findings for this section of the investigation are inferential. 3. Findings Instructors remarked upon the immediacy of being able to check in and see how students were progressing on placement to be a huge benefit. Identifying deficits in students’ understanding was a useful feature and students who appeared disengaged from their work placement became apparent early on, thus allowing for early intervention and support. You could see that they were engaging with the course material regularly rather than leaving them to it for three months (Paula, Instructor) I was able to pinpoint exactly where the learning deficit was within the online discussion. But also it became very apparent that [the student] wasn’t reading nor were they engaging while they were on placement (Kim, Instructor) The discussion boards were accessed through the Blackboard Learning Management System. Despite the convenience of accessing, the boards with a number of devices (mobile phone, Irish Journal of Technology Enhanced Learning 5 tablet, laptop, PC) instructors and students remarked that the interface was unappealing, cumbersome and rudimentary compared to other applications. The discussions were described as ‘contrived’ at times, with one respondent describing how they felt compelled to engage to adhere to the instructional brief, rather than adding value to the discussion. I was like…I’ll reply to her, and then I’ll go in and reply to him, and then I was like, I wish they’d hurry up and write something… (Lorna, Student) Group dynamics are an important feature of discussion boards. The random allocation of students to groups affected their engagement, participation and the quality of the discussions. Students admitted if they did not know or did not like members of their group, this affected their interest and motivation to participate. Students described discussion boards as ‘useful’, ‘accessible’, and ‘reliable’ as assessment and expressed a preference for using them over the paper-based portfolios they had replaced. However, they were not appropriate and effective in how they were delivered in this assessment. The four listed components emerged from the shortfalls identified by instructors and students in this research. Mitigants are suggested, and led to the development of a for- purpose instructional guide to set up and deliver discussion boards for work-based placement. These four components are Time Management, Moderation, Instructional Guidance and Peer- learning. 3.1 Time Management Instructors need to be cognisant of and plan for the front-end time it takes to set up discussion boards as well as their time commitment to provide feedback. Students might be encouraged to create a group agreement to avoid participants waiting days for a reply as the asynchronous feature frustrated some users. 3.2 Moderation Moderation serves to keep a focus on the discussion, ensures all participants contribute, and maximises the opportunity for collaborative learning. The lack of an appointed moderator and instructor feedback was a considerable drawback to the acceptability of the discussion boards in this research. The absence of a moderator to prompt, encourage users to elaborate on their opinions or ask less participative users questions resulted in some feelings of frustration and apathy towards the task. Despite this, the research identified how some students were proactive communicators and others were more responsive. 3.3 Instructional Guidance No negative commentary was recorded regarding the availability and quality of technical support, and the assignment brief was deemed clear and comprehensible. However, the completion instructions were haphazard, unrefined and unsuitable for this type of activity. Considerations include whether grades will be allocated, or whether the discussion will be assignment-related to offer an opportunity to share ideas. If participation is voluntary, a strategy for encouraging participation is needed. Moderation is recommended with instructional support provided if students are appointed as moderators. 6 3.4 Peer Learning Discussion boards facilitate peer learning. Instructor and student comments regarding peer learning reaffirmed the importance of a grouping strategy to maximise opportunities for peer learning. Suggestions included allocating groups according to friendships to dissuade users being excluded or posting last-minute comments. Alternatively, allocating group members according to the remit of their placement organisation was also suggested - students could be tasked with solving a service-related issue and invited to reflect on the process afterwards. 4. Conclusion COVID-19 social distancing regulations impelled a global focus on blended learning strategies in higher education. This research was conducted prior to the pandemic against a markedly different educational landscape. Digitalisation of education has been a main trend in the 21st century, and much of the research focuses on the perception and attitudes of instructors and learners towards technologies and their implementation. Despite this, this is the only research documenting social care placement students’ experiences of using discussion boards on work-based placement. This research created baseline data on the topic of reflection using discussion boards in an Irish social care placement context. Whilst the process was not entirely effective as assessment in this case, a number of useful findings emerged for future use to enhance the process for instructors and users. Students stated that the experience was valuable to help them understand different professional territories of social care work, and as a result, they benefitted from having a more practical knowledge of the sector. They commented that the process compelled them to engage more frequently in conversations about their placement experiences, regularised their engagement with reflection and helped them to understand other-person perspectives. Four components essential to prepare and deliver a best practice approach for using discussion boards for work based placement emerge from this research. this context, yet there is scope for further investigations into this area. Ironically, these components are rooted in communication and effective time management at each phase of delivery. Discussion Board instructors need to anticipate the front-end set-up expectations and manage their time effectively around the provision of feedback. Clear and unambiguous instructional guidance should be communicated to the student users. Time for moderation must also be considered. Other findings from this study (not reported here) suggest a more time effective strategy is to appoint students as moderators in rotation with requisite training, thus facilitating peer learning, imparting more autonomy to students, and modelling a students as partners in assessment approach. Whilst reflection is ultimately a personal activity, this research found that discussion boards mitigated the sometimes-described isolation of the work-based placement experience. More research using discussion boards in this context is recommended to maximise their functional capacity in supporting learners to understand reflection and prompt active engagement and deeper learning from their placement experiences. Irish Journal of Technology Enhanced Learning 7 References Bates, A. T. (2005). Technology, e-learning and distance education. London: Routledge. https://doi.org/10.4324/9780203463772 Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. https://doi.org/10.1037/13620-004 Caldarola, R. A. (2014). The Effectiveness of Online Discussion Forums on Course Outcomes. Global Education Journal, 2014(1). https:// doi:10.1097/ceh.0000000000000252 CORU (2017). Social Care Workers Registration Board Standards of proficiency for social care workers. Retrieved from: https://www.coru.ie/files-education/scwrb-standards-of- proficiency-for-social-care-workers.pdf [accessed 06 June 2022] de Lima, DP., Gerosa, MA., Conte, TU., & de M Netto, JF. (2019). What to expect, and how to improve online discussion forums: the instructors' perspective. Journal of Internet Services and Applications, 10(1), 1-15. https://doi.org/10.1186/s13174-019-0120-0 Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. Educational psychologist, 26(3-4), 325-346. https://doi.org/10.1080/00461520.1991.9653137 Donnelly, R., & Gardner, J. (2011). Content analysis of computer conferencing transcripts. Interactive learning environments, 19(4), 303-315. https://doi.org/10.1080/10494820903075722 Douglas, T., James, A., Earwaker, L., Mather, C., & Murray, S. (2020). Online discussion boards: Improving practice and student engagement by harnessing facilitator perceptions. Journal of University Teaching & Learning Practice, 17(3), 7. https://doi.org/10.53761/1.17.3.7 Giacumo, L. A., Savenye, W., & Smith, N. (2013). Facilitation prompts and rubrics on higher‐order thinking skill performance found in undergraduate asynchronous discussion boards. British journal of educational technology, 44(5), 774-794. https://doi.org/10.1111/j.1467-8535.2012.01355.x Guldberg, K., & Pilkington, R. (2007). Tutor roles in facilitating reflection on practice through online discussion. Journal of Educational Technology & Society, 10(1), 61-72. https://doi.org/10.1111/j.1365-2729.2006.00171.x Henri, F. (1992). Computer conferencing and content analysis. In Collaborative learning through computer conferencing (pp. 117-136). Berlin: Springer. https://doi.org/10.1007/978-3-642-77684-7_8 https://doi.org/10.4324/9780203463772 https://doi.org/10.1037/13620-004 https://www.coru.ie/files-education/scwrb-standards-of-proficiency-for-social-care-workers.pdf https://www.coru.ie/files-education/scwrb-standards-of-proficiency-for-social-care-workers.pdf https://doi.org/10.1080/00461520.1991.9653137 https://doi.org/10.53761/1.17.3.7 https://doi.org/10.1111/j.1467-8535.2012.01355.x https://doi.org/10.1111/j.1365-2729.2006.00171.x https://doi.org/10.1007/978-3-642-77684-7_8 8 Hurst, B., Wallace, R., & Nixon, S. B. (2013). The impact of social interaction on student learning. Reading Horizons, 52(4), 5. Retrieved from: https://scholarworks.wmich.edu /cgi/viewcontent.cgi?article=3105&context=reading_horizons Ixer, G. (1999). There's no such thing as reflection. The British Journal of Social Work, 29(4), 513-527. https://doi.org/10.1093/bjsw/29.4.513 Kay, R.H. (2006). Developing a comprehensive metric for assessing discussion board effectiveness. British Journal of Educational Technology, 37,761-783. https://doi.org/10.1111/j.1467-8535.2006.00560.x Kember, D., Leung, D. Y., Jones, A., Loke, A. Y., McKay, J., Sinclair, K., & Yeung, E. (2000). Development of a questionnaire to measure the level of reflective thinking. Assessment & evaluation in higher education, 25(4), 381-395. https://doi.org/10.1080/713611442 Kohn, A. (2011). The case against grades. Educational Leadership, 69 (3), 28-33. Retrieved from: http://www.montessoriprivateacademy.com/wp-content/uploads/2013/01/alfie-kohn- article-from-educational-leadership.pdf Kolb, D. A. (1984). Experiential Learning: Experience as the source of learning and development. New Jersey: Prentice Hall. Kuepper-Tetzel, C. E., & Gardner, P. L. (2021). Effects of temporary mark withholding on academic performance. Psychology Learning & Teaching, 20(3), 405-419. https://doi.org/10.177.147525721999958 Loncar, M., Barrett, N. E., & Liu, G. Z. (2014). Towards the refinement of forum and asynchronous online discussion in educational contexts worldwide: Trends and investigative approaches within a dominant research paradigm. Computers & Education, 73, 93-110. https://doi.org/10.1016/j.compedu.2013.12.007 Lucas, U., & Tan, P. (2006). Assessing levels of reflective thinking: the evaluation of an instrument for use within accounting and business education. In 1st Pedagogic Research in Higher Education Conference, Liverpool Hope University, Liverpool (pp. 2-3). [online], available: https://www2.uwe.ac.uk/faculties/BBS/BUS/Research/DRC/prhe.pdf [accessed 17 June 2022] Perkowska-Klejman, A., & Odrowaz-Coates, A. (2019). Measuring the unmeasurable? Differences in reflexive thinking among Polish students. The New Educational Review, 55, 77-88. https://doi.org/10.15804/tner.2019.55.1.06 Ransdell, S. (2013). Meaningful posts and online learning in Blackboard across four cohorts of adult learners. Computers in Human Behaviour, 29(6), 2730-2732. https://doi.org/10.1016/j.chb.2013.07.021 Russell, T. (2013). Has Reflective Practice Done More Harm than Good in Teacher Education? Phronesis, 2(1), 1-99. https://doi.org/10.7202/1015641ar https://doi.org/10.1093/bjsw/29.4.513 https://doi.org/10.1111/j.1467-8535.2006.00560.x https://doi.org/10.1080/713611442 http://www.montessoriprivateacademy.com/wp-content/uploads/2013/01/alfie-kohn-article-from-educational-leadership.pdf http://www.montessoriprivateacademy.com/wp-content/uploads/2013/01/alfie-kohn-article-from-educational-leadership.pdf https://doi.org/10.177.147525721999958 https://doi.org/10.1016/j.compedu.2013.12.007 https://www2.uwe.ac.uk/faculties/BBS/BUS/Research/DRC/prhe.pdf https://doi.org/ https://doi.org/10.1016/j.chb.2013.07.021 https://doi.org/10.7202/1015641ar Irish Journal of Technology Enhanced Learning 9 Salmon, G. (2003).E-moderating: The key to teaching and learning online(2nd ed.), London :Routledge Falmer. https://doi.org/10.4324.9780203465424 Salmon, G. (2014). Gilly Salmon, The Five-Stage Model. Retrieved from: https://www.gillysalmon.com/five-stage-model.html Sumsion, J., & Fleet, A. (1996). Reflection: can we assess it? Should we assess it? Assessment & Evaluation in Higher Education, 21(2), 121–130. https://doi:10.1080/0260293960210202 Taradi, S. K., & Taradi, M. (2004). Expanding the traditional physiology class with asynchronous online discussions and collaborative projects. Advances in Physiology Education, 28(2), 73-78. https://doi.org/10.1152/advan.00017.2003 Tsingos, C., Bosnic-Anticevich, S., Lonie, J. M., & Smith, L. (2015). A model for assessing reflective practices in pharmacy education. American journal of pharmaceutical education, 79(8). https://doi.org/10.5688/ajpe798124 Vygotsky, L. (1978). Interaction between learning and development. Readings on the development of children, 23(3), 34-41. Wormeli, R. (2006). Accountability: Teaching through assessment and feedback, not grading. American secondary education, 14-27. Xie, K., Di Tosto, G., Lu, L., & Cho, Y. S. (2018). Detecting leadership in peer-moderated onlinecollaborative learning through text mining and social network analysis.Internet and HigherEducation,38,9–17. https://doi.org/10.1016/j.iheduc.2018.04.002 Yip, K. S. (2005). Self-reflection in reflective practice: A note of caution. British Journal of Social Work, 36(5),777-788. https://doi.org/10.1093/bjsw/bch323 https://www.gillysalmon.com/five-stage-model.html https://doi:10.1080/0260293960210202 https://doi.org/10.1152/advan.00017.2003 https://doi.org/10.5688/ajpe798124 https://doi.org/10.1016/j.iheduc.2018.04.002 “How are you getting on?” Evaluating discussion boards as assessment for social care work-based placement Abstract 1. Introduction 1.1 Communication and Constructivism 1.2 Factors of Engagement 1.3 Reflection 3. Findings 3.1 Time Management 3.2 Moderation 3.3 Instructional Guidance 3.4 Peer Learning 4. Conclusion