Australasian Journal of Educational Technology, 2022, 38(1). 53 Enhancing online protocols through design-based research to improve cognitive presence in a large enrolment course Janet Mannheimer Zydney University of Cincinnati, United States of America Aimee deNoyelles, Baiyun Chen University of Central Florida, United States of America Kerry Rich Patton University of Cincinnati, United States of America Instructors face challenges in facilitating higher levels of shared cognition in large enrolment classes. One strategy to foster shared cognition is the use of asynchronous discussions; however, these can be difficult to support with large numbers of students. Online protocols have been found to help students take more ownership of the discussion, reducing the workload of instructors and thus may be helpful in this context. The purpose of this study was to determine the most effective design of online protocols in large enrolment classes. The study used a design-based research methodology to iteratively design, assess and refine the online protocols and the design principles of its underlying protocol pedagogy. Participants for this study were 1,286 students enrolled in a blended undergraduate class in business that used online protocols for discussions over three semesters. The paper describes how iterative changes in the design of protocol-based discussions influenced students' cognitive presence. Students’ perceptions of cognitive presence were significantly higher in the second and third iterations, and the concepts shared by group members were significantly more integrated by the third iteration. Findings suggest that with careful design, these enhanced protocols are a potentially useful strategy to facilitate asynchronous online discussions in large classes. Implications for practice or policy: • Students need additional incentives to actively participate in large enrolment courses. • Instructors can use peer questioning to increase shared cognition in large enrolment courses. • Instructors should create concise directions with modelling of exemplary posts to help reduce confusion. • Students in large enrolment courses benefit from additional scaffolding of norms to foster a sense of trust. • Design-based research provides an effective methodology to examine both theoretical and practical implications of online protocols on cognitive presence. Keywords: blended learning, business education, cognitive presence, large enrolments, online discussions, protocols, design-based research Introduction There is a trend in higher education of increasing class sizes due to calls for expanded student access, tightening budgets and administrative pressure (Hornsby & Osman, 2014; Lynch & Pappas, 2017). One mechanism to increase enrolments is to use blended or online components. Regardless of modality, large classes are prone to issues with class participation (Lynch & Pappas, 2017; Rocca, 2010), student interaction (Lynch & Pappas, 2017; Orellana, 2006) and the development of higher-level cognition (Monks & Schmidt, 2011; Mulryan-Kyne, 2010). Thus, it is important to devise strategies to increase students’ participation and interaction, as well as enhance shared cognition in large enrolment courses. One successful method in creating more interaction in online or blended environments is through asynchronous discussions (Gao et al., 2013). However, it can be challenging to facilitate meaningful online discussions to enhance shared cognition (Gao et al., 2013), especially in large enrolment courses when the instructor cannot carefully monitor the discussion. One potential strategy to overcome this challenge is to Australasian Journal of Educational Technology, 2022, 38(1). 54 use a discussion protocol. A protocol identifies a clear purpose for the discussion, gives participants instructions on their roles within the discussion, describes how those roles interact with one another and provides specific time frames for those interactions (McDonald et al., 2012). Protocols for online discussions in small graduate courses have been found to increase group cognition as well as enable students to take ownership of their discussion, thereby reducing the workload of the instructor (Zydney et al., 2012). In previous research, B. Chen et al. (2017) found protocol-based discussions to be a plausible teaching strategy in classes with large enrolments. Although this is a positive and useful result, with limited evidence, there is more work to be done to identify the most optimal design for using protocols for asynchronous discussion within large enrolment courses. To systematically explore the optimal design of protocol-based discussions in large enrolment courses, we employed a design-based research (DBR) methodology for this study. DBR is defined as: A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually sensitive design principles and theories. (Wang & Hannafin, 2005, pp. 6–7) The purpose of using this methodology was to enhance the design principles of a protocol pedagogy within the context of large enrolment courses. This study consisted of three iterations of a protocol-based discussion format implemented in an undergraduate, large enrolment, blended business course over 3 semesters, with the specific intent of answering the following research questions: • How did students’ feedback influence the design improvements of the protocol-based discussions? • How did the improvements in the design of the protocol-based discussion affect the level of cognitive presence and students’ perception of cognitive presence? Literature review This section explores the intersection of the literature concerning large enrolment classes, cognitive presence and a protocol pedagogy. Large enrolment classes The definition of a large enrolment class varies depending on the level, discipline and modality of a course. For undergraduate foundational classes, research tends to draw evidence from classes with more than 100 students (Maringe & Sing, 2014). Most empirical studies on large enrolment courses focus on face-to-face classes, with limited research on online or blended modalities (Owston et al., 2019). Research suggests that the ideal size of online classes is below 30 students (D’Orio, 2017), with 12 to 16 to achieve the most optimal level of interaction (D’Orio, 2017; Orellana, 2006). Unfortunately, the massification of higher education has increased class size to be over the ideal size. Several teaching strategies have shown promise to enhance learning and compensate for the loss of personal interaction time for large enrolment classes. In particular, small-group discussion forums can enhance interactivity (Kim, 2013); and well-designed discussion protocols can promote higher perceptions of social, cognitive and teaching presences in large classes (B. Chen et al., 2017). Based on the limited research available, further research to investigate the implementation of discussion protocols in large online classes is needed. Cognitive presence Cognitive presence, situated within the community of inquiry (CoI) framework, is the ability to create and confirm meaning through sustained communication and reflection (Garrison & Arbaugh, 2007). The cognitive presence discourse consists of four phases: putting forth an issue (triggering event), sharing ideas about the issue (exploration), connecting ideas (integration) and developing solutions to those issues (resolution) (Garrison et al., 2001). A discussion activity reaches a high level of cognitive presence when students complete these four phases (Garrison & Arbaugh, 2007). Australasian Journal of Educational Technology, 2022, 38(1). 55 A review of the literature reveals that online discussions often do not reach levels of cognitive presence beyond the exploration phase (Y. Chen et al., 2019; Garrison & Arbaugh, 2007; Sadaf & Olesova, 2017). This means that while students may be exchanging information and ideas, they are rarely connecting and expanding on ideas or applying new ideas to other contexts. A low level of cognitive presence can be explained in part by the design of these discussions not requiring students to progress to the integration and resolution phases (Garrison & Arbaugh, 2007). To address this need, this study adopted a strategy based on a protocol pedagogy that combines several instructional approaches associated with increased cognitive presence, as described in the next section, in order to provide the additional structure that students need in large enrolment courses. Protocol pedagogy A protocol pedagogy is an instructional approach for fostering student-centred discussions, consisting of four design principles: “(a) enabling active participation through varied roles, (b) creating equity through structure, (c) fostering trust through establishing norms and (d) prompting connections with texts” (Zydney et al., 2020, p 2). Each of these principles is enacted through a protocol, which can be defined as a collaborative tool for fostering meaningful discussions. Each discussion protocol includes a discussion goal, roles for participants to play, rules for how the roles should interact and a time frame (McDonald et al., 2012). Online protocols have shown promising results, perhaps because they integrate several successful strategies for online discussions (deNoyelles et al., 2014; DiPasquale & Hunter, 2018). To create active participation, protocols assign specific roles to play during the discussion (Hancock & Rowland, 2017; Hou, 2011; Olesova et al., 2016). They are very structured, which is common to many effective online discussion strategies (deNoyelles et al., 2014; Hung et al., 2015; Sadaf & Olesova, 2017; Schindler & Burkholder, 2014). Protocols also establish norms by specifying rules for interaction as found in other methods, such as setting the number, length and deadline of postings (Ergulec, 2019), requiring a specific type of response to a discussion prompt (Schindler & Burkholder, 2014) or providing built-in sentence starters (Nachowitz, 2018). Finally, they prompt connections through several methods, such as peer review and feedback (Ekahitanond, 2013; Hancock & Rowland, 2017; Nachowitz, 2018). To illustrate how the four principles of a protocol pedagogy are enacted through an online protocol, each principle will be described in relation to an example protocol called the "tuning protocol”, created by Allen and McDonald (1993). First, to enable active participation, a protocol includes clearly defined roles, such as presenter, critical friend and devil’s advocate. Each of these roles serves a different purpose to make an active, intentional contribution to the overall goal of the conversation. For example, in the online version of the tuning protocol (McDonald et al., 2012), there are two roles: presenter and participant. Second, to create equity, each protocol provides structured rules and time frames for interaction. In an online tuning protocol, the protocol typically spans 2 weeks. The presenter posts their work at the start of the first week. At the end of the first week, the participants post their feedback to the presenter. Then, in the middle of the second online week, the presenter writes a reaction to the feedback received. The second online week wraps up with everyone debriefing the discussion. Third, to foster trust and risk-taking, protocols establish norms for participants to follow. These specific rules for interaction help ensure participants feel more secure in sharing their ideas (McDonald., 2012). In the tuning protocol, the participant reflects on how they would like to contribute feedback to the presenter. Then the participant posts warm feedback (strengths, appreciation) and cool (considerations, suggestions) feedback to the presenter, often guided by provided sentence starters (e.g., “I wonder if …”, “Have you thought about …”). Fourth, a protocol prompts connections with texts by giving participants a meaningful collective goal and by providing time for reflection. Goals for discussion may vary widely depending on course goals (e.g., brainstorming, problem- solving, offering feedback). In the tuning protocol, the presenter and participant collaboratively determine an action plan to improve the work. Method This section explains the DBR process used, participants and context, instruments and the analysis used in this study. Australasian Journal of Educational Technology, 2022, 38(1). 56 DBR methodology Over three semesters (Spring 2014, Fall 2014 and Spring 2015), we used a DBR methodology to systematically explore through an iterative process both the theoretical and practical implications of using a protocol-based pedagogy within the context of a large enrolment course. Our research team consisted of two instructional designers, located at the university where the data was collected, who consulted with the instructor on all aspects of the design of the protocols in the course, along with two researchers from another institution conducting similar research and focused on the data analysis. We had all previously engaged in DBR studies as well as the design of online discussions. This study was the culmination of three phases of educational design research: analysis and exploration, design and construction, and evaluation and reflection (McKenney & Reeves, 2012). The goal of the analysis and exploration phase was to begin to understand the problem through examining the literature and meeting with various stakeholders. During this phase, we (the two instructional designers) met with the instructor of the course to understand the problem of creating a community in large enrolment courses. In the design and construction phase, the focus was to develop a tentative solution to the problem. During this phase, we collaborated with the instructor to co-create an initial design for the online discussions based on the protocol pedagogy, which is described in the Design decisions for iterations section. The evaluation and reflection phase involved iteratively testing and refining the design over three iterations. This phase of the DBR study is the main emphasis of this paper, and the outcomes of this iterative testing can be found in the Results and Discussion sections. Participants and context The context for the study was an undergraduate introductory business course taught over 3 semesters by the same instructor at a large south-eastern university in the United States of America. The course was for students interested in starting a new business. Even though the course was offered in the College of Business, students of sophomore standing across disciplines could register for it. The class was classified as a blended learning course as it consisted of both weekly face-to-face lectures that were streamed and recorded and online activities, such as asynchronous discussions. There were three graded protocol-based discussions, which focused mainly on having students post their work and provide feedback to one another. After each semester, we collaborated with the instructor of the course to create a new iteration of the protocol-based discussions based on feedback from the prior semester. Students in the class had the option to participate in an anonymous research survey for extra credit. There was an alternate extra credit assignment for those who opted not to take the survey. In total, 1,286 students responded to the survey A breakdown of the participants is shown in Table 1. Table 1 Number of participants by iteration (N = 1,286) Iteration No. Iteration 1 394 Iteration 2 446 Iteration 3 446 In all three iterations, students participated in discussion groups of approximately 10 students (called “Bazinga circles”). Students were randomly assigned to groups by their teacher, and then three groups were randomly selected, one from each iteration, to provide a representative sample of the larger population. Group formation remained the same throughout the semester. Research instruments and data analysis At the end of each semester, we collected data through feedback surveys and by downloading discussion posts of the selected groups, under a protocol approved by the university’s Institutional Review Board. Australasian Journal of Educational Technology, 2022, 38(1). 57 Feedback survey The feedback survey consisted of 12 Likert-scale questions taken from the extensively tested CoI survey (Arbaugh et al., 2008) on the construct of cognitive presence and two open-ended questions to determine what to improve in the next iteration. The Likert-scales ranged between 1 and 5, with 1 being strongly disagree and 5 being strongly agree. Frequencies were used to analyse students’ perceived levels of cognitive presence with reference to the protocol-based discussions. A Welch’s ANOVA test and a Games- Howell post-hoc test were conducted to investigate the differences in perceptions between the three iterations. The open-ended questions asked participants to offer their perceptions of the protocol-based discussions. We used a directed content analysis approach to interpret the open-ended survey comments through the lens of the principles contained within a framework (protocol pedagogy, in this case). Hsieh and Shannon (2005) suggested that this is a useful choice if “existing theory or prior research exists about a phenomenon that is incomplete or would benefit from further description” (p. 1281). Expanding upon the underlying theory of the protocol pedagogy, especially with relation to the unique context of a large enrolment setting, was the goal of this study. Themes were generated from Iteration 1, then 2, then 3, and then examined to detect overlap or new developments. Discussion post coding We analysed the discussion posts for indicators of cognitive presence. The coding was done using the cognitive presence element of a modified CoI coding template (see Table 1) as established and validated by Zydney et al. (2012). Table 2 CoI coding template (adapted from Garrison et al., 2000) Elements Categories Indicators Cognitive Presence Triggering Event Voluntarily brings up new or related topic Exploration - Individual Repeating or describing ideas from text, describing experiences and prior knowledge Exploration - Group Exchange of ideas (clarifying questions or ideas, confirmation of understanding, repeating ideas, agreement with ideas) Integration - Individual Connecting prior knowledge or experience to text, connecting texts together Integration - Group Connecting other participant’s ideas, expanding on someone else’s idea, disagreement, or counterpoints Resolution - Individual Applying ideas to your own future, theory to practice Resolution Applying what has been discussed Reprinted from Computers & Education, Vol 58 /edition 1, Zydney et al., Creating a community of inquiry in online environments: An exploratory study on the effect of a protocol on interactions within asynchronous discussions, Page 82, Copyright (2012), with permission from Elsevier. The modified COI coding template distinguished between individual and group categories within cognitive presence. Individual cognitive presence referred to participants’ postings that were like monologues unrelated to other participants’ posts, and group cognitive presence referred to posts where participants were engaged in discussion with other participants (Zydney et al., 2012). We (the two researchers) individually coded the three groups’ discussions. First, we identified the number of ideas within a post. And, then we coded each idea for the highest level of cognitive presence. A total of 553 ideas were coded with 71 discrepancies that resulted in an initial inter-rater reliability of 87%. Then, we met to reconcile variations in coding. All discrepancies were 100% resolved. After the discussion posts were coded, we calculated frequencies and percentages for each category by individual and group presence to analyse the differences among the three iterations. In addition, the Fisher’s exact test (2-sided) for small sample size contingency tables was used to identify statistically significant results and post-hoc comparisons. Following this analysis, we identified examples of posts that illustrated the quantitative differences found between iterations. Australasian Journal of Educational Technology, 2022, 38(1). 58 Results This section describes how student feedback influenced the design enhancements of a protocol-based discussion as well as the how students’ perception and level of cognitive presence changed as a result. Design decisions for iterations The purpose of the protocol-based discussions was to engage the students to improve their work together; therefore, the tuning protocol, as described earlier, was selected. Table 3 illustrates how each of the four design principles of the protocol pedagogy (Zydney et al., 2020, p. 2) were applied to the initial design of a discussion. Table 3 Application of a protocol pedagogy’s design principles (Zydney et al., 2020, p. 2) to a discussion on a business model Design principles Application to discussion-based activities Principle 1: Enabling active participation through varied roles Directions asked everyone to take on two roles during the discussion: presenter and reviewer. Everyone completed both roles. Principle 2: Creating equity through structure Instructions provided deadlines for posts and responses. Directions asked everyone: • to post a business model and request feedback on areas of concern. • to reply to at least one other person with feedback. Principle 3: Fostering trust through establishing norms Norms on how to give constructive feedback were established by prompting three types of feedback: • Warm: describe what you see is good, or what you like about their business model. • Cool: consider aspects of the work that might be improved or clarified. Ask a question of this person in your post, such as “Have you thought about …” • Hard: ask deeper questions that get at the larger aspects of this person’s model. Principle 4: Prompting connections with texts Connections with texts were fostered by the collective goal that everyone would provide text-based feedback on the business models. Table 4 provides a summary of the design changes that occurred across the three iterations. Table 4 Instruction modifications made to the discussion activity Design element Iteration 1 Iteration 2 Iteration 3 Instruction on feedback prompts X X Example posts that modelled feedback X X Simplify instruction focus X X All parts of the discussion resided in one forum X X Reminders for due dates provided X X Rubric used to evaluate quality of discussion X X Grade for discussion increased from 1% to 5% of final course grade X X Note. X = presence of design element First iteration design and feedback In the first iteration, each discussion was separated into two online discussion forums: one forum was for Part A, in which students posted their business model; the other forum was for Parts B and C. In Part B, students provided feedback to at least one other student’s business model, and in Part C, the presenter reflected on how the peer feedback influenced the development of their business model. Then the presenter submitted the revised business model to the instructor as an individual assignment. Australasian Journal of Educational Technology, 2022, 38(1). 59 When students accessed the Part A discussion forum, they received instructions about both the discussion as well as the subsequent assignment. The Appendix includes the instructions pertinent to the protocol- based discussion about business models and illustrates how those instructions changed for each iteration. Students were provided with guidance on how to provide warm, cool and hard feedback to their peers. The due dates were provided in the written instructions. The due dates for Parts A and C appeared in students’ calendars, but Part B did not, since only one due date could be specified within the same forum. The discussion was worth 1% of the total course grade. The discussion was graded based solely on participation. Overall, the open-ended survey feedback from students in Iteration 1 was positive; for instance, “The discussions in this course were extremely helpful as a hands-on approach to understanding the course material”. However, they largely reported confusion about the due dates. For example, one student noted: “When Parts B and C were due on different dates but in the same discussion [forum], only the latest due date showed up on the assignments page or grades page, so I ended up missing Part B, because I thought nothing was due that week”. They also felt that having the multiple parts of the discussion in separate forums was confusing. A student commented, “It was hard to understand the feedback without the original post”. Students found the directions lengthy and sometimes confusing. A student shared, “Having the assignments and discussions grouped as one assignment was confusing. I think emphasizing that discussions and assignments are separate will be more beneficial to the organization of the course in future semesters”. Although comments about peer feedback were generally positive, several mentioned not receiving feedback about their initial post; for instance, “Some classmates [in my group] did not participate and it hindered the purpose of the discussion”. A few suggested that making the discussion worth more points would prompt more attention as well as improve the quality of the response. Second iteration design and feedback The following changes were made in Iteration 2: • Instructions on how to give warm, cool and hard feedback were removed as the instructor felt they were too complicated. • Instead, exemplary posts that modelled warm, cool and hard feedback were provided. The instructor felt that students often benefitted from seeing previous examples. • To improve the course organisation, the instructions for the discussion and associated assignment were separated. • To streamline the structure of the discussions, the discussion forums were consolidated into one forum. • To provide better reminders for students, due dates were added to the course calendar for each part of the discussion. • Grading was clarified by adding a rubric for the discussion. The rubric had two criteria: relevance of the posts to the discussion and quality of the feedback. • The discussion went from 1% to 5% of the total course grade in order to encourage greater participation. After Iteration 2, the feedback survey included fewer comments that indicated confusion about the due date or instructions, indicating that the design change had been effective. The exemplary posts included in the instructions in Iteration 2 were generally lauded by students. One student remarked in the feedback survey, “The examples given were the biggest help to me in completing the assignments”. Another elaborated, “I really enjoyed the structured discussions and I appreciated the examples at the end of the instructions, because if I felt a little confused or lost with the directions, the examples made it very clear”. Students in Iteration 2 indicated in the feedback survey that they enjoyed reviewing the work of others and receiving feedback. One student praised, “The structured discussions really helped me understand my business model to the fullest. I was always interested to see what my other Bazinga members posted and what feedback they had for me” . The issue of not getting a response was mentioned a few times, although at a much lower rate than in Iteration 1. For those who did get responses, more critical peer feedback was craved. One mentioned, “The structured discussion comments were typically bland and unhelpful. I believe it was because people were afraid of offending other people”. Australasian Journal of Educational Technology, 2022, 38(1). 60 Third iteration design and feedback The third and final iteration was designed with the intention of focusing on the need for more critical peer responses, which was voiced in the second iteration feedback. It was the same as the second with one change: the instructions on providing warm and cool feedback were added back in, with the intention that it would enhance the quality of peer feedback, as well as enable students to give more critical feedback with less concern about offending others. The cool feedback prompted students to ask each other questions, such as “Have you thought about …?” Although the quality of the feedback improved in Iteration 3, there was still room for improvement. For example, in the feedback survey, while some commented along the lines of “I received a lot of great feedback from my Bazinga circle teammates which helped me progressively improve on every assignment”, others had comments similar to “I didn’t get back what I put into the discussions and I had to extrapolate one-sentence responses into something I felt was relevant”. Changes in perception of cognitive presences over three iterations We calculated the cognitive presence score as the average rating of all cognitive presence questions (Arbaugh et al., 2008). Since all questions used a 5-point Likert scale, the average cognitive presence score of each iteration could range between 0 and 5. Analyses of the survey findings (Table 5) show that for all three iterations the perceptions of cognitive presence were high. In particular, students’ perceptions of cognitive presence were significantly higher in the second and third iterations. Table 5 Means and standard deviations of cognitive presence perceptions n Means SD Minimum Maximum Iteration 1 394 3.74 0.75 1.00 5.00 Iteration 2 446 4.07 0.64 1.00 5.00 Iteration 3 446 4.07 0.65 1.33 5.00 Total 1,286 3.97 0.70 1.00 5.00 Since the Levene’s F test revealed that the homogeneity of variance assumption was not met (p = .002), a Welch’s F test was used. There was a statistically significant difference among three iterations as determined by a one-way ANOVA, Welch’s F(2, 835) = 32.07, p < .001, indicating that not all iterations had the same cognitive presence score. The estimated omega squared (ω2 = .05) suggested a small effect size (Cohen, 1988), indicating that approximately 5% of the total variation in students’ perception of cognitive presence is attributable to differences among the three iterations. The Games-Howell post-hoc procedure was conducted to determine which differences were significant among the three iterations. The post-hoc test revealed that the improved discussion protocol in Iteration 2 (p < .001) and 3 (p < .001) did have an effect on students’ perception of cognitive presence. However, there was no statistically significant difference between iterations 2 3 (p = 1.0). Specifically, when the discussion protocol was enhanced with feedback prompts, models, simplified instruction, consolidated forums, due date reminders, grading rubrics and increased grade percentage, students perceived a higher sense of cognitive presence. Additional instruction on feedback prompts did not further increase students’ perception of cognitive presence. Changes in levels of cognitive presence in small-group discussions We further analysed the discussion posts to determine whether there were subtle changes in levels of cognitive presence as a result of the design decisions. Significant differences were found in individual cognition (p = .04) and group cognition (p = .02), as shown in Table 6. Australasian Journal of Educational Technology, 2022, 38(1). 61 Table 6 Differences in categories within cognitive presence Element Iteration 1 Iteration 2 Iteration 3 Frequency % Frequency % Frequency % Triggering event 0 0 0 0 0 0 Individual Exploration 25 52 14 29 24 44 Integration 7 15 5 10 2 4 Resolution 3 6 5 10 0 0 Sub-total 35 73 24 49 26 47 Group Exploration 6 10 9 18 8 15 Integration 7 17 14 29 19 35 Resolution 0 0 2 4 2 4 Sub-total 13 27 25 51 29 53 Total 48 100 49 100 55 100 Individual cognition tended to decrease, and group cognition tended to increase with each iteration, indicating that students were discussing the concepts more as a group. The greatest improvement in individual cognition followed design changes made after Iteration 1. Post-hoc Fisher’s exact test (2-sided) comparisons of frequencies of individual cognitive presence indicated significant improvement between Iterations 1 and 2 (p = .03) and Iterations 1 and 3 (p = .04) but not between Iterations 2 and 3 (p = 1.0). Within individual cognition, the only element that differed significantly among the iterations was individual resolution (p = .05). Post-hoc comparisons revealed that the decrease between Iterations 2 and 3 was significant (p = .03); whereas the differences between the other iterations was not. Here is an example of a discussion post in Iteration 2 coded as individual resolution: We [co-workers from student’s business] have changed the direction of how to approach the business. Instead of self-publishing, we are going to publish the book with the help from an organization and provide the link to the membership website on the back of the book. This response demonstrates the individual resolution level through applying concepts to future development. Within group cognition, the greatest improvement was also following the design changes made after Iteration 1. Using the same procedure as mentioned above, post-hoc comparisons revealed significant differences between Iterations 1 and 2 (p = .03) and Iteration 1 and 3 (p = .01), whereas there was no significant improvement between Iterations 2 and 3 (p = .71). The only element within group cognition that differed significantly among the iterations was group integration (p = .05). Group integration posts allowed members to connect and expand on other participants’ ideas: An issue to think about are start up costs, the cost to run while building your clientele as well as other ways of advertisement. Perhaps advertising in the newspaper or handing out flyers would reach those who may not have social media or do not get on often. Post-hoc comparisons revealed significant improvement in group integration between Iterations 1 and 3 (p = .03) and no other significant differences between other iterations. Thus, in this case, the additional changes of adding back the directions after Iteration 2 were needed to see significant differences in group integration. This difference may be related to the fact that students in Iteration 3 tended to ask more questions for others to consider than in prior iterations. Discussion Each of the sections that follow describe the modified design principles of the protocol pedagogy in relation to the study’s findings and prior research, ending with practical implications and suggestions for future research. Australasian Journal of Educational Technology, 2022, 38(1). 62 Incentivising active participation through varied roles The first design principle was modified from enabling active participation through varied roles to incentivising active participation through varied roles. Encouraging active participation in a large enrolment class is not an easy task to achieve (Hornsby & Osman, 2014). One strategy is having participants play different roles within the discussion (Hancock & Rowland, 2017; Zydney et al., 2020). Two roles were set up for this discussion activity: presenters and reviewers. In the first iteration, even though each student was required to provide at least one feedback comment to another student, a number of students did not participate. As a result, some students did not receive any feedback. Since each discussion activity was only worth 1% of the total course grade, some students suggested that increasing the weight of the assignment could potentially incentivize active participation. Therefore, the instructor increased the credit for the activity to 5% of the total course grade in the second and third iteration, and the situation improved. Practical recommendations for incentivising active participation when utilising protocols within large enrolment classrooms include the following: increase the percentage of the total course grade associated with the discussion activities, increase the number of replies required, set up at least two distinct roles and model different roles with explicit examples. Future research could explore protocol activities with increased number of replies required and additional roles. We speculate that increasing the number of roles and replies required by students could potentially increase the number of perspectives one can receive. Creating equity through a streamlined structure The second design principle was expanded from creating equity through structure to creating equity through a streamlined structure. We speculate that the protocol structure (e.g., small groups, set time frames, well- defined rules for interaction) might help by making sure everyone participates equitably in the discussion (McDonald et al., 2012), which can be a challenge in large enrolment classes that tend to have lower levels of student participation (Kim, 2013). However, when protocol structures were too complex, students became confused and ended up missing posts and due dates. This resulted in some students not receiving feedback on their work, which made the experience feel inequitable. On the other hand, streamlining the directions too much can come at a risk of students missing elements. For example, without the directions that specifically instructed students to ask each other questions in the second iteration, many students did not include this in their feedback, which may have limited group cognition. Prompting students with sentence starters to challenge one another’s thinking may help increase collaborative knowledge building (Nachowitz, 2018). Once the prompt to ask questions within the cool feedback was added back into the directions during the third iteration, the level of group cognition significantly improved. This peer feedback may have helped promote higher levels of critical thinking, corroborating findings in a study by Ekahitanond (2013). Practical recommendations for creating more streamlined structures when utilising protocols within large enrolment classrooms include the following: consolidate the same discussion into one forum, simplify the directions for the protocol, add models for students to follow and include a more structured rubric. In particular, providing models of example posts enabled the instructor to reduce the directions needed to explain the steps of the protocol, and a number of students commented on how helpful these example posts were in enabling them to complete the assignment. Although the findings indicate that the structure of protocols needs to be streamlined within the context of large enrolment classes, more research and experience are needed to determine the right level of detail for protocol directions. Fostering trust through scaffolded norms The third design principle was expanded from fostering trust through establishing norms to fostering trust through scaffolded norms. It is essential to clearly spell out the norms, but they can be more effectively understood and implemented when there are some additional scaffolds in place. Modelling exemplary discussion posts is an example of scaffolds which have helped to establish norms. Similar to what was reported in the Schindler and Burkholder (2014) literature review, we found that these examples gave students a model in which to work from. Providing norms for how to provide feedback can also help foster trust. In the final iteration, targeted information about how to provide warm and cool feedback was added back in, which included a couple of Australasian Journal of Educational Technology, 2022, 38(1). 63 sentence starters (“Have you considered ….?”). These sentence starters may help a student who is more novice in the discipline more securely and effectively critique someone else’s ideas. Despite the added scaffolds in the final iteration, there were still some students who said they did not get a meaningful response. Additional sentence starters may be needed to further help peers with critical, honest feedback. Practical recommendations for fostering trust through scaffolded norms when utilizing protocols within large enrolment classrooms include the following: showcase a model post that explicitly critiques another person’s work with effective cool feedback, provide a wider variety of sentence starters to give peers additional guidance in how to critique (Ekahitanond, 2013) and change the norms so that each person is expected to reply to at least two others, which would likely ensure that each person gets at least one response. Future research could include a more focused look at the effect of critical sentence starters on cognitive presence, as well as the effect of requiring more interaction within the discussion. Prompting connections through questioning The fourth design principle was expanded from prompting connections with texts to prompting connections through questioning. The protocol initially designed to help students make connections to the text was expanded to provide instructions that modeled questioning responses which result in a higher level of interaction. This finding supports prior research which showed that the types of questions asked can result in increased cognitive presence (Olesova et al., 2016). Inclusion of modelling questioning responses can also increase the effectiveness of group discussions (Choi et al., 2008). Our findings indicate the modelling questions increased student interaction, which may have resulted in an increase in group cognition. Practical implications for prompting connections within a large enrolment class include the following: provide concise instructions that prompt students to ask questions and offer models of example feedback to increase discussion interaction and responses. Each of these design principles may provide a discussion structure to encourage connections through effective constructive responses among group members and increase cognitive presence throughout the discussion. Although the results show that including questioning responses in feedback increased interaction and group cognition, further research is needed to determine the type and amount of questioning examples that are effective. Summary of implications Large classes often have challenges with class participation (Lynch & Pappas, 2017; Rocca, 2010), student interaction (Lynch & Pappas, 2017; Orellana, 2006) and cognitive presence (Monks & Schmidt, 2011; Mulryan-Kyne, 2010; Sadaf & Olesova, 2017). This DBR study found that the use of online protocols helps to improve these factors. Although we followed the established educational design research methodology by McKenney and Reeves (2012), the study was not without its limitations. First, no student demographic information was collected. The research questions focused on the three iterations of design; therefore, other differences among these groups were beyond the scope of the study. Second, due to time constraints, it was not possible to code all the discussion posts in the three iterations of this large enrolment course. We randomly selected three representative sample groups and coded 553 ideas for the three groups of discussions. Third, the student samples changed over the three iterations. However, we considered the three samples similar as students were all sophomores interested in taking the same class. In spite of the limitations noted, the study contributes to both the theoretical and practical implications of implementing a protocol pedagogy within a large enrolment class. From a theoretical perspective, it expands upon the design principles from the literature on protocols. Australasian Journal of Educational Technology, 2022, 38(1). 64 Table 7 Comparison of original design principles (Zydney et al., 2020, p. 2) to enhanced design principles Original design principles Enhanced design principles Principle 1: Enabling active participation through varied roles Principle 1: Incentivising active participation through varied roles Principle 2: Creating equity through structure Principle 2: Creating equity through a streamlined structure Principle 3: Fostering trust through establishing norms Principle 3: Fostering trust through scaffolded norms Principle 4: Prompting connections with texts Principle 4: Prompting connections through questioning As shown in Table 7, when a protocol pedagogy is used within large enrolment classes, the design principles were enhanced to include additional incentives, more streamlining, increased scaffolding and the use of questioning. The finding also has several practical implications for the design of large enrolment courses using protocols. One recommendation repeated in all the design principles was the importance of modelling, which can be accomplished by providing authentic examples (i.e., exemplary student posts), similar to what was noted in Schindler and Burkholder’s (2014) literature review on critical thinking in online discussions. Modelling allows for instruction to be streamlined, helps show students how to take on varying roles within the discussion, establishes norms for how to give effective feedback and illustrates peer questions to enhance connections. However, despite the modelling provided, there were still students who felt the feedback was lacking. One recommendation is for the instructor to explicitly model how to give effective constructive feedback and to provide additional sentence starters for giving feedback, as was also noted by Ekahitanond (2013) in a study on critical thinking in online discussions. Another practical recommendation that cut across multiple design principles was to simply increase the required number of replies and/or number of roles that students take on in the discussion. Increasing these requirements not only makes the discussion more active but also increases the number of connections made, which helps address the difficulty in fostering active participation in large enrolment classes noted by Hornsby and Osman (2014). Requiring at least two replies also increases the likelihood that students will receive feedback, enhancing their trust in feeling that they will get back what they put into the discussion. Along with increasing the requirements, we recommend increasing the percentage of the grade to provide additional incentive for participation. An additional recommendation common to several design principles is the need for concise directions that prompt peer questioning. Peer questioning – a technique recommended by Choi et al. (2008) – appeared to increase the number of connections students made and, as a result, the level of shared cognition. Overall, this study found that a less-is-more approach is effective for large enrolment courses where students are more likely to skim the directions. Offering clear directions in the fewest words possible with examples that illustrate instructor expectations is desirable. These enhanced protocols may provide a mechanism to bridge the research on strategies to improve critical thinking within online discussions to work within large enrolment settings. References Allen, D., & McDonald, J. P. (1993). Keeping student performance central: The New York assessment collection. Coalition of Essential Schools, Brown University. Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the community of inquiry framework Using a multi-institutional sample. Internet and Higher Education, 11(3–4), 133– 136. https://doi.org10.1016/j.iheduc.2008.06.003 Chen, B., deNoyelles, A., Zydney, J. M., & Patton, K. (2017). Creating a community of inquiry in large enrollment online courses: An exploratory study on the effect of protocols within online discussions. Online Learning Journal, 21(1), 165–188. https://doi.org10.24059/olj.v21i1.816 Chen, Y., Lei, J., & Cheng, J. (2019). What if online students take on the responsibility: Students’ cognitive presence and peer facilitation techniques. Online Learning, 23(1), 37–61. https://doi.org10.24059/olj.v23i1.1348 http://dx.doi.org/10.1016/j.iheduc.2008.06.003 http://dx.doi.org/10.24059/olj.v21i1.816 http://dx.doi.org/10.24059/olj.v23i1.1348 Australasian Journal of Educational Technology, 2022, 38(1). 65 Choi, I., Land, S. M., & Turgeon, A. (2008). Instructor modeling and online question prompts for supporting peer-questioning during online discussion. Journal of Educational Technology Systems, 36(3), 255–275. https://doi.org10.2190/ET.36.3.c Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum. deNoyelles, A., Zydney, J. M., & Chen, B. (2014). Strategies for creating a community of inquiry through online asynchronous discussions. Journal of Online Learning and Teaching, 10(1), 153–165. https://jolt.merlot.org/vol10no1/denoyelles_0314.pdf DiPasquale, J., & Hunter, W. J. (2018). Critical thinking in online asynchronous discussions: A systematic review. Canadian Journal of Learning and Technology, 44(2), 1–25. https://doi.org10.21432/cjlt27782 D’Orio, W. (2017, May 17). One size doesn’t fit all. Inside Higher Ed. https://www.insidehighered.com/digital-learning/article/2017/05/17/online-class-sizes-one-size- doesnt-fit-all Ekahitanond, V. (2013). Promoting university students’ critical thinking skills through peer feedback activity in an online discussion forum. Alberta Journal of Educational Research, 59(2), 247–265. https://jmss.org/index.php/ajer/article/view/55617 Ergulec, F. (2019). Design and facilitation strategies used in asynchronous online discussions. Malaysian Online Journal of Educational Technology, 7(2), 20–36. https://doi.org10.17220/mojet.2019.02.002 Gao, F., Zhang, T., & Franklin, T. (2013). Designing asynchronous online discussion environments: Recent progress and possible future directions. British Journal of Educational Technology, 44(3), 469–483. https://doi.org10.1111/j.1467-8535.2012.01330.x Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2, 87–105. https://doi.org10.1016/S1096-7516(00)00016-6 Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1). https://doi.org10.1080/08923640109527071 Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future directions. The Internet and Higher Education, 10(3), 157–172. https://doi.org10.1016/j.iheduc.2007.04.001 Hancock, C., & Rowland, B. (2017). Online and out of synch: Using discussion roles in online asynchronous discussions. Cogent Education, 4(1), 1–19. https://doi.org10.1080/2331186X.2017.1368613 Hornsby, D. J., & Osman, R. (2014). Massification in higher education: Large classes and student learning. Higher Education, 67(6), 711–719. https://doi.org10.1007/s10734-014-9733-1 Hou, H. (2011). A case study of online instructional collaborative discussion activities for problem- solving using situated scenarios: An examination of content and behavior cluster analysis. Computers & Education, 56(3), 712–719. https://doi.org10.1016/j.compedu.2010.10.013 Hsieh, H., & Shannon, S., (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288. https://doi.org10.1177/1049732305276687 Hung, W., Flom, E., Manu, J. & Mahmoud, E. (2015). A Review of the instructional practices for promoting online learning communities. Journal of Interactive Learning Research, 26(3), 229–252. https://www.learntechlib.org/p/130598/ Kim, J. (2013). Influence of group size on students’ participation in online discussion forums. Computers & Education, 62, 123–129. https://doi.org10.1016/j.compedu.2012.10.025 Lynch, R. P., & Pappas, E. (2017). A model for teaching large classes: Facilitating a “small class feel.” International Journal of Higher Education, 6(2), 199–212. https://doi.org/10.5430/ijhe.v6n2p199 Maringe, F., & Sing, N. (2014). Teaching large classes in an increasingly internationalising higher education environment: Pedagogical, quality and equity issues. Higher Education, 67(6), 761–782. https://doi.org1007/s10734-013-9710-0 McDonald, J. P., Zydney, J. M., Dichter, A., & McDonald, B. (2012). Going online with protocols: New tools for teaching and learning. Teachers College Press. McKenney, S., & Reeves, T. (2012). Conducting educational design research. Routledge. Monks, J., & Schmidt, R. M. (2011). The impact of class size on outcomes in higher education. The B.E. Journal of Economic Analysis & Policy, 11(1). https://doi.org/10.2202/1935-1682.2803 Mulryan-Kyne, C. (2010). Teaching large classes at college and university level: Challenges and opportunities. Teaching in Higher Education, 15(2), 175–185. https://doi.org10.1080/13562511003620001 http://dx.doi.org/10.2190/ET.36.3.c https://jolt.merlot.org/vol10no1/denoyelles_0314.pdf http://dx.doi.org/10.21432/cjlt27782 https://www.insidehighered.com/digital-learning/article/2017/05/17/online-class-sizes-one-size-doesnt-fit-all https://www.insidehighered.com/digital-learning/article/2017/05/17/online-class-sizes-one-size-doesnt-fit-all https://jmss.org/index.php/ajer/article/view/55617 http://dx.doi.org/10.17220/mojet.2019.02.002 http://dx.doi.org/10.1111/j.1467-8535.2012.01330.x http://dx.doi.org/10.1016/S1096-7516(00)00016-6 http://dx.doi.org/10.1080/08923640109527071 http://dx.doi.org/10.1016/j.iheduc.2007.04.001 http://dx.doi.org/10.1080/2331186X.2017.1368613 http://dx.doi.org/10.1007/s10734-014-9733-1 http://dx.doi.org/10.1016/j.compedu.2010.10.013 http://dx.doi.org/10.1177/1049732305276687 https://www.learntechlib.org/p/130598/ http://dx.doi.org/10.1016/j.compedu.2012.10.025 http://dx.doi.org/1007/s10734-013-9710-0 https://doi.org/10.2202/1935-1682.2803 http://dx.doi.org/10.1080/13562511003620001 Australasian Journal of Educational Technology, 2022, 38(1). 66 Nachowitz, M. (2018). Scaffolding progressive online discourse for literary knowledge building. Online Learning, 22(3), 133–156. https://doi.org10.24059/olj.v22i3.1261 Olesova, L., Slavin, M., & Lim, J. (2016). Exploring the effect of scripted roles on cognitive presence in asynchronous online discussions. Online Learning, 20(4), 34–53. https://doi.org10.24059/olj.v20i4.1058 Orellana, A. (2006). Class size and interaction in online courses. The Quarterly Review of Distance Education, 7(3), 229–248. http://wps.prenhall.com/wps/media/objects/4512/4621309/Survey_Online_Class_Size.pdf Owston, R., York, D. N., & Malhotra, T. (2019). Blended learning in large enrolment courses: Student perceptions across four different instructional models. Australasian Journal of Educational Technology, 35(5), 29–45. https://doi.org10.14742/ajet.4310 Rocca, K. A. (2010). Student participation in the college classroom: An extended multidisciplinary literature review. Communication Education, 59(2), 185–213. https://doi.org10.1080/03634520903505936 Sadaf, A. & Olesova, L. (2017). Enhancing cognitive presence in online case discussions with questions based on the Practical Inquiry model. American Journal of Distance Education, 31(1), 56–69. https://doi.org10.1080/08923647.2017.1267525 Schindler, L. A., & Burkholder, G. J. (2014). Instructional design and facilitation approaches that promote critical thinking in online discussions: A review of the literature. Higher Learning Research Communications, 4(4), 11–29. https://doi.org10.18870/hlrc.v4i4.222 Wang, F. & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23. https://doi.org10.1007/BF02504682 Zydney, J. M., deNoyelles, A., Seo, K. (2012). Creating a community of inquiry in online environments: An exploratory study on the effect of protocols on interactions within asynchronous discussions. Computers & Education, 58(1), 77–87. https://doi.org/10.1016/j.compedu.2011.07.009 Zydney, J. M., Warner, Z., & Angelone, L. (2020). Learning through experience: Using design based research to redesign protocols for blended synchronous learning environments. Computers & Education, 143, Article 103678. https://doi.org/10.1016/j.compedu.2019.103678 Corresponding author: Janet Mannheimer Zydney, janet.zydney@uc.edu Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Zydney, J. M., deNoyelles, A., Chen, B., & Patton. K. R. (2022). Enhancing online protocols through design-based research to improve cognitive presence in a large enrolment course. Australasian Journal of Educational Technology, 38(1), 53-68. https://doi.org/10.14742/ajet.5994 http://dx.doi.org/10.24059/olj.v22i3.1261 http://dx.doi.org/10.24059/olj.v20i4.1058 http://wps.prenhall.com/wps/media/objects/4512/4621309/Survey_Online_Class_Size.pdf http://wps.prenhall.com/wps/media/objects/4512/4621309/Survey_Online_Class_Size.pdf http://wps.prenhall.com/wps/media/objects/4512/4621309/Survey_Online_Class_Size.pdf http://dx.doi.org/10.14742/ajet.4310 http://dx.doi.org/10.1080/03634520903505936 http://dx.doi.org/10.1080/08923647.2017.1267525 http://dx.doi.org/10.18870/hlrc.v4i4.222 http://dx.doi.org/10.1007/BF02504682 https://doi.org/10.1016/j.compedu.2011.07.009 https://doi.org/10.1016/j.compedu.2019.103678 mailto:janet.zydney@uc.edu https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5994 Australasian Journal of Educational Technology, 2022, 38(1). 67 Appendix: Iterative changes made to directions Key Plain text – All iterations Strikethrough text – Removed in Iteration 2 Italicised text – Added in Iteration 2 Underlined text = Added in Iteration 3 For this group discussion posting, I would like you to use a specific structure called a “Tuning” discussion. The Tuning process allows for reflection and meaningful feedback about your Business Model, and will likely result in your creating a better business model, and earning a better grade too. This structure assumed that you want to improve your business model, and that your Bazinga Circle members will deliver thoughtful and substantive feedback. Think of it like turning up a car or an orchestra. Your communal goal is to help each other make A’s on this assignment, so do this early in the week so you can use their feedback to improve your success on your assignment. There are three parts to this discussion structure, Parts A, B, and C. Part A Briefly remind your Bazinga Circle members about your business concept. Share your business model with them. Provide enough detail to accurately describe the business model, but please keep it concise. Try to make your post as readable as possible. Next, read over your business model and think about particular aspects that you would like to improve, such as Key Activities, for example. At the end of your discussion post, ask the group to consider at least one aspect of your business model about what you would like their constructive feedback. Focus on aspects that will help improve your business model. For example, you might ask your group members how your proposed solution could be more doable in the real world. This might be an aspect of your model you would like to improve, and allows your group members to focus on that particular aspect and offer you some meaningful feedback. Ask your Circle members to review your submission and give you feedback about something specific in your Business Model you believe could be improved (For example, perhaps you want their feedback on your Revenue Stream or on Validation and Pivoting). [Example removed for space considerations] Part B Read through your Bazinga Circle’s discussion posts and choose at least one person to whom you will reply. Note that everyone in your Circle should receive feedback, so reply to someone who has not received a reply yet. There are three kinds of feedback that can be utilized to improve your feedback to this member of your Circle: Warm: describe what you see is good, or what you like about their business model. Cool: consider aspects of the work that might be improved or clarified. Ask a question of this person in your post, such as “Have you thought about…” Hard: ask deeper questions that get at the larger aspects of this person’s model. Keeping these three kinds of feedback in mind, offer some constructive feedback that focuses on aspects of their business model that they posed at the end of their discussion post. Your communal goal is to help each other make A’s on this assignment, so pay close attention to the deadlines. You are required to list one thing that you liked, and one thing you believe could be improved AND suggestions for improving it. Australasian Journal of Educational Technology, 2022, 38(1). 68 Here are some suggestions to provide more helpful feedback to others. There are two kinds of feedback that can be utilized to improve your feedback to Circle members: Warm – describe what you see is good, or what you like about their Value Proposition. Cool – consider aspects of their Business Model that might be improved or clarified. Ask a question of this person in your post, such as “Have you thought about…?” Keeping “warm” and “cool” feedback in mind, offer some constructive feedback that focuses on aspects of their post that they posed at the end of their discussion.” For Example: "I am focusing on giving feedback to Catherine! I really love Cat's idea and think she did a great job describing her business model canvas. It was precise and would give potential investors a good understanding of her business idea very clearly. But have you thought about maybe talking to more people about your concept and getting more feedback? Particularly someone in the field of travel would be very beneficial to improving your business model! I love that Cat's idea has found a niche in an already established industry and I think that is very important. However, I think something Cat needs to ask herself is how she is going to get her business out there so people know of this new kind of service? How will she stand out against her competition and let her consumers know of her new innovation and how she can stay different from everyone else in the travel field. Cat asked for feedback on her revenue estimates and I believe her revenue stream would be a usage fee because her customers would be using her particular service that she has created. I think in order to improve your business concept Cat must use more of the terminology from lecture to help make the understanding of her concept even better!" Part C For the final part of this structured tuning discussion, return to the first post you wrote about your own business model and consider the feedback you have received from your group member(s). In your post, discuss your answers to these questions: 1. What news ideas or questions have been raised about your work thanks to this feedback? 2. How has this feedback influenced the next steps in improving your business model? Reflect on what happened as a result of this structured discussion. What suggestions did your group member(s) provide that helped you to improve your Business Model? Introduction Literature review Large enrolment classes Cognitive presence Protocol pedagogy Method DBR methodology Participants and context Research instruments and data analysis Feedback survey Discussion post coding Results Design decisions for iterations First iteration design and feedback Second iteration design and feedback Third iteration design and feedback Changes in perception of cognitive presences over three iterations Changes in levels of cognitive presence in small-group discussions Discussion Incentivising active participation through varied roles Creating equity through a streamlined structure Fostering trust through scaffolded norms Prompting connections through questioning Summary of implications References Appendix: Iterative changes made to directions