Australasian Journal of Educational Technology, 2016, 32(5). 35 The evolution and evaluation of an online role play through design-based research Elizabeth A. Beckmann and Sango Mahanty Australian National University This paper presents selected findings from a 5-year design-based research case study of the evolution of an online role play that allows postgraduate students to explore the complexities inherent in land rights negotiations between indigenous peoples and others. In the context of Laurillard’s (2002) conversational framework and a design-based research methodology, diverse private and public discussion forum spaces were created for group negotiations on a learning management system (LMS) platform. Our analysis of the conversational framework structure in the evolved role play showed that all four stages – discursive, adaptive, integrative, and reflective – were evidenced, with the adaptive and integrative stages cycling through multiple times. The online role play, whilst implemented as a simple virtual world, facilitated personal, deep and socialised learning experiences focused on consultation, negotiation and decision-making. We also found that student anonymity was not necessary for full engagement in role play, and that students chose to incorporate communication technologies outside the LMS into their learning activities. This research shows that with a strong pedagogical design, and attention paid to an evidence-based iterative improvement cycle, online role plays can provide powerful collaborative learning experiences. Introduction A role play is an activity that requires students to assume a specific identity (either a named historical character or a character designated by occupation, nationality, culture, worldview, or any combination of these traits); research the views/experiences that identity could be expected to have; and operate in character within the role play to solve problems and explore issues, usually with other characters (Jones, 2007; Wills et al., 2009; Wills, Leigh, & Ip, 2011). Educators have long documented the capacity of face-to-face role plays to facilitate understanding of complex issues (Ellington, 2004; Teahan, 1975). Putting role plays online removes geographical and time barriers to participation in these powerful learning opportunities (Freeman & Capper, 1999; McLaughlan & Kirkpatrick, 2014; Russell & Shepherd, 2010; Wills et al., 2009; Wills et al., 2011). Providing synchronous communication opportunities for students who are not co-located enables a faster pace of interaction, and facilitates the communicative collaboration that fosters active learning (Ip, Linser, & Naidu, 2001). Asynchronous interactions meet the needs of students who are in different time zones or unable to be online at specific times, and, importantly, gives them more time to think about relevant theories and models as they build their role play strategies. Role plays are particularly valuable as intensive opportunities for interactive engagement and learning, especially among professionals (Harris, Cornelius, & Gordon, 2009; Maier, Baron, & McLaughlan, 2007). Ertmer et al. (2010) argue that, in professional education, a role play scenario that allows students to collaborate on a “potentially real situation” (p. 73) fosters enhanced critical reasoning, and helps new professionals start thinking “more like seasoned practitioners” (p. 90). This rationale was the driving force behind the creation in 2009 of the online role play RISOS (which means “resources” in the Melanesian language Tok Pisin) for postgraduate students of development studies at an Australian university. Previous research (Beckmann, 2010; Beckmann & Kilby, 2010) had shown that the relevant student cohort typically included a mix of full-time, part-time, on-campus and off-campus enrolments from within and outside Australia (especially from, or working in, the Asia-Pacific region), with at least one third speaking English as a second or subsequent language. Only an online activity could bridge the diverse barriers to participation, and the opportunities to provide both synchronous and asynchronous communication meshed with our desire to engage all students in a discursive, conversational approach to learning (Laurillard, 2002). While many researchers distinguish between role plays, simulations, and games (e.g., Ellington, 2004; Russell & Shepherd, 2010), RISOS has elements of all three. It has specific roles that students must adopt; it has the “tactical decision” (essentially quantitative) and “social process” (essentially qualitative) nature of a simulation (Gredler, 1992); and it displays the key features of a game – namely overt competition for Australasian Journal of Educational Technology, 2016, 32(5). 36 an objective, and constraints (rules) within which players must operate (Ellington, 2004). To emphasise this focus on objectives and rules, RISOS is described as a game to students, but has all the characteristics of an online role play as defined by Wills et al. (2009, p. 10): [An activity] designed to increase understanding of real life human interaction and dynamics—participants assume a role in someone else's shoes or in someone else's situation; do authentic tasks in an authentic context; [and are involved in] substantial in-role human interaction such as collaboration, negotiation, [and] debate [with] interaction between roles … substantially in an online environment. Over 6 years, RISOS evolved from a 2-hour, face-to-face, non-assessed course activity available only to on-campus students into a substantive, intellectually demanding 8-week online role play that contributes 70% to the course grade. RISOS allowed on- and off-campus students to participate on an equal footing as they explore the complex social, cultural, political and economic dimensions of resource negotiations between indigenous peoples and representatives of civil society, the state and the private sector. In this paper, we first describe how, from 2009 to 2014, we used the pedagogical design perspectives afforded by the conversational framework (Laurillard, 2002), within the context of an iterative design-based research (DBR) perspective (Reeves, Herrington, & Oliver, 2005), to guide the pedagogical and technological evolution of RISOS. We then report on outcomes, including a brief analysis in terms of the conversational framework and three key findings related to the technological design – how we used discussion forum permissions in a learning management system (LMS) to create diverse conversational pathways in a simple virtual world, the high level of engagement of students with their roles despite a lack of anonymity, and the need to accommodate the dynamic nature of students’ use of communication technologies. Design decisions Funding, legal and logistical constraints meant we could not create RISOS within an existing sophisticated virtual world environment, nor embrace significant development costs (although an internal teaching grant did give us access to a short period of technological support). Cornelius, Gordon, and Harris (2011, p. 58) suggest that, because “the performances and interactions of [role play] participants help to shape the context, enabling participants to play their roles effectively may … be more important than providing a realistic environment”. Naidu, Ip, and Linser (2000) argue that a dynamic goal-based learning scenario becomes a highly motivational learning activity if the goal is of intrinsic interest to the learners, who eagerly seek the knowledge and skills to pursue that goal. Like Westera (2011), we believed that credibility was more important than realism, so we designed RISOS as a dynamic goal-based scenario (Schank, Fano, Bell, & Jona, 1994; Naidu et al., 2000) set in a development context, in which each learner has a goal (mission) associated with the role they assume. Learners must thus focus not only on playing a specific role, but also on trying to achieve an outcome consistent with the worldview of that role, usually by working towards objectives that (in character) they have themselves defined, as individuals or in groups. The RISOS scenario is set in a fictitious district in a named developing country: as we sought opportunities for students to engage fully with their online identity and role (Ertmer et al., 2010), locating RISOS in a real country was crucial for the authenticity of cultural contexts, knowledge and implications. In this district, three fictitious indigenous villages (Klostu, Namel and Tudak) must negotiate with the GDP mining company, the Green Forest Logging Company, and the Blue Planet Conservation non-government organisation (NGO) to manage the natural environment and promote community welfare. Each of the six groups has its own distinct objectives, known only to the group, which may or may not overlap with those of other groups. Each student takes on the role of either a villager or a company/NGO representative. The two teaching academics also participate fully. One acts as the game’s “Expert”, providing authentic inputs, such as daily media posts and responses from government representatives or politicians that the villagers or others wish to consult or influence. The quality and authenticity of this expert input derives from the academic’s internationally recognised real-life expertise in the field and with the specific country in question. The other academic acts as “Game Manager/Facilitator”, supporting students in their roles and tasks, mediating inter-group discussions where necessary, and dealing with practical issues, such as problems with technologies. Australasian Journal of Educational Technology, 2016, 32(5). 37 While many students were already experienced development professionals, others were fresh from diverse undergraduate degrees. As we were asking students from diverse professional, educational, cultural and national backgrounds to explore problems that are inherently complex, have many perspectives, and no right answers, it was crucial that we had a pedagogical infrastructure that created equitable opportunities for group work and shared learning over a limited time period. We were influenced here by Bateson’s (1972) work on second and third order learning – respectively, teaching students how to learn, and helping learners to uncover the limitations in their own knowledge systems and engage with the existence of multiple perspectives. This led us to the conversational framework (Laurillard, 2002), specifically distilled in the context of online learning, as a structure and conceptual guide for both educational design and analysis. The conversational framework includes four stages against which to review RISOS design and outcomes (Figure 1). In the discursive stage, after the teacher presents a new concept, the learners enter into a dialogue with the teacher, questioning and clarifying the idea and its language. In the interactive stage, the learners put the new concept into practice as they interact with teacher-constructed tasks, and receive feedback on their performance. In the adaptive stage, learners attempt to put their understanding into practice, modifying and adapt their actions in the light of what they have learned, and begin to construct their own links between ideas. Finally, there is a reflective stage in which both learners and teachers relate theory back to the practice, adjust their thinking in the light of reflection on their learning, and consider what future actions might be more successful. Moreover, Laurillard (2009) argues that the conversational framework can be used not only to specify the requirements for an optimally effective technology-enhanced learning experience, as we had done in our design process, but also to test the lived experience of students using that design, using a set of pertinent questions (Figure 2). Figure 1. The learning process as a conversational framework (after Laurillard, Stratfold, Luckin, Plowman, & Taylor, 2000) Australasian Journal of Educational Technology, 2016, 32(5). 38 Do [the learning activities] motivate students to: • access explanations and presentations of the theory, ideas or concepts? • ask questions about their understanding of the theory, etc., by providing the opportunity for answers from the teacher, or their peers? • offer their own ideas and conceptual understanding, by providing comment on them from the teacher, or their peers? • use their theoretical understanding to achieve a clear task goal by adapting their actions in the light of their understanding, or in response to comments or feedback? • repeat practice, by providing feedback on actions that enables them to improve performance? • repeat practice, by enabling them to share their trial actions with peers, for comparison and comment? • reflect on the experience of the goal-action-feedback cycle, by offering repeated practice at achieving the task goal? • discuss and debate their ideas with other learners? • reflect on their experience, by having to articulate or produce their ideas, reports, designs, performances, etc. for presentation to their peers? • reflect on their experience, by having to articulate or produce their ideas, reports, designs, performances, etc., for presentation to their teachers? Figure 2. Checklist of questions to analyse effectiveness of a conversational framework learning design (Laurillard, 2009, p. 19) To facilitate each stage of the conversational framework (Laurillard, 2002), and in line with the principles of constructive alignment (Biggs & Tang, 2011), we created authentic learning activities and assessment tasks (Herrington, Reeves & Oliver, 2010) within the goal-based scenario that would allow the students to achieve three complex learning outcomes through their participation in RISOS. Thus, we wanted students to learn to identify the key factors impacting on resource negotiations and development outcomes for lands owned by indigenous people; to understand the potential imperatives and responses of indigenous people and other actors/stakeholders in resource management and development, through critical analyses of the social, economic and political contexts; and to gain a detailed awareness of the nature of negotiation processes, compromises and outcomes. With regards to students’ technology needs, we knew that the off-campus students often had limited access to high bandwidth Internet (although this issue was decreasing over time), and many students were using mobile learning devices (an increasing proportion over time). Given the global reach of the course, we needed to provide maximum flexibility in terms of synchrony and asynchrony to account for differences in work commitments, time zones, preferences for learning and communication devices, and variable Internet access. We therefore had to work with technology that was fundamentally adaptable, robust and easy to use from a student’s prospective, and relatively easily managed and inexpensive from the technical standpoint. Research methodology In designing and refining RISOS, we wanted to be rapidly responsive to learners’ needs and concerns, and consistently use these to shape an increasingly more effective technology-enhanced learning experience. We also sought to contribute to the research literature around long-term observations of complex online role plays, complementing, for example, the literature on the award-winning Mekong e-Sim, which, like RISOS, focuses on complex decision-making (Maier et al., 2007; McLaughlan & Kirkpatrick, 2014). We therefore scoped RISOS as a 5-year case study using the conversational framework (Laurillard, 2002) for both dialogic design and analysis. To ensure that all changes were driven by students’ behaviour and feedback as learners and technology users, we adopted a DBR methodology (Bannan-Ritland, 2003; Reeves et al., 2005; Wang & Hannafin, 2005) that would allow for continuous improvement cycles (Bessant & Francis, 1999). DBR is a systematic but flexible research methodology that is specifically aimed at improving educational practices through “iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings” (Wang & Hannafin, 2005, p. 6). Dede, Ketelhut, Whitehouse, Breit, and McCloskey (2009) and Anderson and Shattuck (2012) are among many authors who have concluded that DBR methodology provides best practice in effective, theoretically informed and evidence-based development of complex learning environments. Australasian Journal of Educational Technology, 2016, 32(5). 39 By allowing for pragmatic, contextualised, and diverse inputs grounded both theoretically and authentically (Design Based Research Collective, 2003; Wang & Hannafin, 2005), DBR met our risk management criteria because it supports “the impact, transfer, and translation of education research into improved practice” (Anderson & Shattuck, 2012, p. 16). Moreover, DBR has been identified as particularly suited to research on technology in higher education, especially when – as with RISOS – the focus is to: Emphasise content and pedagogy rather than technology, give special attention to supporting human interactions and nurturing learning communities, modify the learning environments until the pedagogical outcome is reached, and reflect on the process to reveal design principles that can inform … future … projects. (Reeves et al., 2005, p. 110) The key to DBR methodology is iteration, both in feedback cycles and in effective modifications based on that feedback. We therefore implemented DBR data collection activities and cycles from 2009 to 2014, using diverse student feedback mechanisms (approved by the university’s Human Research Ethics Protocol 2009/222). We refined these over time as course design became more robust so that we were not over- surveying the students. In this paper, we use data and quotes from five sources of identified or anonymous data collected from 2010 to 2014. First, we use identified feedback on design and technological issues collected via the course’s Technical Help (TechH) forum which remained open throughout each RISOS iteration. This space allowed students (in or out of character) to identify concerns or queries, request support, or view tool guides (e.g., how to use wikis to record group ideas and construct group documentation; how to use real-time communication technologies). We also use data sourced at the end of each iteration: openly shared feedback from an online debriefing discussion forum, and individual written reflections only seen by the lecturer. To ensure that the design informed by the conversational framework was enabling the intended range of narrative, interactive communicative, and productive aspects of learning, we also analysed students’ role-based forum postings using a checklist (Figure 3) derived from Laurillard’s (2009, p. 19) own analytical questions (Figure 2). We sought at least five to ten forum postings indicating a positive response to each question, with this analysis being carried out by the non-lecturing author (EB), to minimise bias. Finally, we sourced annual anonymous feedback and satisfaction ratings through the university’s standardised course evaluation surveys (2009–2014). Discursive activities • Is there initial discussion between teacher and students as concepts are presented? • Are relevant communication technologies presented and students encouraged to give feedback on these? • Is feedback from students to teachers acted upon? Interactive and adaptive activities • Is a structured discussion environment provided for group members, within and between groups? • Do students have a task-based goal, and are they able to clarify this with teachers as necessary? • Is there is meaningful feedback from the teacher to the student at all times? • Are learners helped to revise their thinking during the learning activities because of the way in which the activities are structured? • Are learners encouraged to adapt their actions in the light of experience (e.g., through within- group and/or between-group discussions)? • Are teachers able to adapt their responses in the light of their reflection on students’ performances during the activities? Reflective activities • Are learners encouraged to reflect on their actions/decisions in the light of experience (e.g., through within-group and/or between-group discussions)? • Are learners encouraged to reflect on RISOS as a whole, as well as on its constituent activities? • Are learners encouraged to reflect on their learning during each phase of RISOS? • Is there is a structured period for reflection, involving both teachers and students? Figure 3. Checklist of questions used to analyse effectiveness of the conversational framework learning design in RISOS (developed from questions in Figure 2, after Laurillard, 2009, p. 19) Australasian Journal of Educational Technology, 2016, 32(5). 40 In this paper, we present a limited selection of our findings, namely those pertaining to the successful implementation of Laurillard’s (2002) conversational framework in the pedagogical design of RISOS as an online activity, and three specific technological design issues: the creation of effective discussion spaces; the relevance of student anonymity online to effective learning; and the increasing role of students, in contrast to educational designers, in deciding which communication technologies are best used to facilitate their collaborative learning. Results and discussion Overall outcomes The RISOS scenario proved sufficiently credible and authentic to allow students to engage with many issues characteristic of development projects. Despite the complexity of roles and inputs, the highly realistic (and therefore often unexpected) twists and turns in policies, events and negotiations led students to report that the game was highly engaging: “I am happy to admit I became totally engrossed in the role at times, to the point where I would forget it was actually a hypothetical situation” (student feedback, 2011). In the 5 years of this case study, students expressed very high satisfaction with assessment design, feedback, and learning activities in the university-standardised post-course evaluation surveys. For example, mean ratings for the statement “Overall, I was satisfied with my learning experience in this course” (where 1 is strongly disagree and 5 is strongly agree) were 4.6, 4.5, 4.4, 4.9 and 4.3 from 2010 to 2014 respectively (with no statistically significant differences between these ratings, given sample sizes). McLaughlan and Kirkpatrick (2014) have suggested that online role plays are particularly relevant to professional education, as they can create intense learning activities focusing on teamwork, decision- making, leadership, communication and negotiation skills. In RISOS, we saw much evidence of exactly those skills being used in students’ engagement and assessment tasks. This was confirmed by an international external peer reviewer invited to benchmark the master’s program that includes RISOS: he reported that the game incorporated “creative teaching mechanisms designed to imbue students with the ability to consider complex issues and problems from multiple perspectives”, entailed “a level of emotional investment and reflection that simply would not be possible in regular written assignments”, and enabled all students to “glean … insights into the complexities of indigenous resource management” (J. Igoe, peer review report, pers. comm., 2011). We believe that incorporating a formal DBR process into the implementation of RISOS as an online role play was crucial in ensuring best practice design. We made four key responses to feedback: more fairly acknowledging the time commitment that students made to RISOS by weighting assessment tasks more heavily (to its current 70% of overall course assessment); providing a clear set of staged deadlines for specific activities and assessable tasks; giving much more emphasis to the debriefing stage, both in time and in assessment value; and acknowledging that students were making their own choices of communicative technologies to achieve the game outcomes, and that we had to accommodate this in assessment approaches (discussed further below). A minimalist virtual world through the use of discussion forums In line with the conversational framework as its guiding pedagogical structure, RISOS required effective opportunities and learning spaces for multiple intra- and inter-group discussions in which students could “articulate a view, re-articulate it in the light of the other’s utterance, ask questions, and reply to questions, though not necessarily synchronously” (Laurillard, 2002, p. 250). The adaptive and constructivist approach of dynamic goal-based learning is particularly well addressed by asynchronous online discussions, as these provide the time and reflective space necessary for each participant not only to act out his or her role, and react to others in their respective roles, but also to acquire, reflect on, and act in accordance with, new information that is relevant to that character, adjusting appropriately to changing situations (Naidu et al., 2000). Our primary technical requirement in designing RISOS was thus to find an effective way of facilitating asynchronous discussions in private and shared spaces between and among diverse groupings of 20 to 30 RISOS participants. In the first year this was done via an externally hosted social networking platform, but this proved untenable for a second year for financial and legal reasons, so in 2010 RISOS moved onto the Australasian Journal of Educational Technology, 2016, 32(5). 41 university’s LMS, Moodle® 1.8 (upgraded via Moodle® 1.9.6 to Moodle® 2.2 in 2012 and Moodle® 2.5 in 2013). By using the Moodle® Groups and Grouping tools, and adjusting individual user permissions on the forum activities, we created a set of text-based forums to allow asynchronous communication at whole- group, inter-group and intra-group levels (Figure 4). Thus, when a student first enters the RISOS site with his/her role already determined, his/her available communication channels are appropriately restricted. • 6 group spaces, one for each group: where members of a specific group share information and develop strategies without the knowledge of other groups. • 3 village-village negotiating spaces: for separated discussions between pairs of villages (Klostu- Namel, Klostu-Tudak, Namel-Tudak). • 9 village-external negotiating spaces: for separated discussions between each village and each external group (e.g., Klostu-GDP, Klostu-Green Forest, Klostu-Blue Planet, repeated for Namel and Tudak). • 6 government-stakeholder discussion spaces: for separated discussions between each village or external group and the Government (e.g., Klostu-Government, GDP-Government). • 3 external-expert discussion spaces: for separated discussions between each external group and the expert (e.g., Klost-Expert; GDP-Expert) • 1 common forum (Deepend District Market): for any group to communicate with any other group, without any privacy. Figure 4. The diversity of communication forums in RISOS created through LMS permissions settings to allow dialogic communication within, between and among groups (as evolved by 2012) Having the TechH forum open to all participants allowed a rapid response to design issues. In the first LMS iteration (2010), for example, a few participants found themselves unexpectedly able to access forums that should have been unavailable to them, and vice versa. These students quickly realised that “espionage” would undermine the game, and, mostly taking the option to remain in character, identified these technical mishaps via the TechH forum: My delight is immense at the opportunity to witness communication between my clanspeople in the distant villages of Klostu and Tudak. … I fear though that if the Big Men in the villages were to learn of my unique listening-in skills that bad things might start to happen in my village. (student with access to other groups’ discussions; TechH forum posting, 2010) I am sure that my family in Klostu is sick with worry about my wellbeing … and no doubt will be trying to contact me … please let me speak with Klostu village. (student without access to own group’s discussion; TechH forum posting, 2010) Similarly, when the paired discussion spaces in the 2010 design proved insufficient to enable all the communication pathways students required, we were quickly alerted to the need for a broader group space, and were able to address this promptly, as shown by the relevant exchange: Student [as Villager]: Klostu, Tudak and Namel have decided to hold a meeting. While the content of this meeting is top secret, we would like an online forum that can accommodate our discussions. Could that be arranged? Game Manager: I've added an Association meeting [forum] which villagers can access until [date] … After that you'll need to revert to your usual bilateral discussions. (TechH forum postings, 2010) We also found that the open TechH forum (used from 2010), where students could answer questions posed by their peers, was more effectively integrated into the role play than the help desk model (used in 2009), where answers were provided by an specialist technologist. The open forum encouraged students to take ownership of the technologies they were using, and helped peer support become central to the game’s infrastructure. For example, until 2012 the LMS settings in individual student profiles influenced how media releases were distributed: either immediately on posting or in a 5pm digest. (After 2012, the 5pm digest became a fixed LMS setting campus-wide). When some 2010 students used the TechH forum to query their failure to receive immediate notification of media releases (which potentially placed them at a Australasian Journal of Educational Technology, 2016, 32(5). 42 disadvantage), their peers quickly responded with appropriate technical advice: “Everyone should make sure that in their [LMS]/RISOS setting, they have YES for track and subscribe” (TechH forum posting, 2010). Anonymity was not necessary to facilitate student engagement Harris et al. (2009) argue that anonymity in online role plays contributes to students’ ability to speak freely in role. Sullivan (2002) and Cornelius et al. (2011) also advocate online anonymity to minimise any potential stereotyping or bias arising from the names or appearances (on photographs) of participants; increase participants’ openness and honesty in online comments; and encourage within-group trust. Those designing online role plays therefore generally build in anonymity: in the Mekong e-sim, for example, students log in using their roles (Maier et al., 2007, p. 1167). Our university’s campus-wide LMS settings, however, meant that RISOS participants could not log in anonymously, and all postings were identified by students’ real names. Moreover, this identification underpinned our capacity to assess participants on indicators of individual engagement and leadership as recorded in the text-based forums. Despite the concerns raised in the literature, however, we did not find any evidence that students’ engagement and learning outcomes had been negatively impacted by the visibility of their real names. Indeed, students responded positively when asked to add real or avatar photos to their user profiles, and the lack of anonymity appeared to foster a sense of common purpose and community: “through our constant interaction [we] created a sort of bond or familiarity with each other even if we've never met in person” (anonymous student feedback, 2010). We recognise that the nature of the RISOS students’ actual or desired professional backgrounds (i.e., as development workers and policymakers) means that many may already have a mature awareness of other world views, an understanding of the complexities and compromises essential in teamwork, and a commitment to community building. These students may thus have been predisposed to negotiate with those of other viewpoints and opinions. They also knew they were all playing arbitrarily assigned roles that did not reflect anything about them as real individuals, which may have helped them be themselves and play a persona at the same time. Certainly we saw that interactions in the role-focused group work led to stronger peer relationships and learning communities, as these student comments from the anonymous evaluation surveys demonstrate: Definitely a sense of community came about through the game. To start with I was unsure whether I would enjoy a group exercise … To my surprise I really got to know my group mates and enjoyed our regular catch ups and also got to know other class mates through the game. It was a fantastic learning experience. (Off-campus student, 2011) This is the only course where I have felt a sense of community amongst students. It was great to engage with off-campus students for the first time and I also enjoyed getting to know the students in my class through various discussions. (On-campus student, 2011) We therefore note that participant anonymity in online role plays may not be such a key requirement as the literature currently suggests. Students decided what online tools to use to support inter-group communication Although focused on group work, ultimately RISOS is about the engagement and learning of individual students. Assessing involvement requires rigorous criteria and processes (outside the scope of this paper) to be applied to records of all discussions: if some peer communications are unrecorded (e.g., on mobile phones or Skype®), gauging an individual’s participation becomes very difficult. Yet it became obvious that, for many students, the LMS’s asynchronous text-based forums were too slow and inadequate for those wanting – and, despite time zone constraints, being accustomed to – more instantaneous and voice-based contact. With each iteration of RISOS, we found students increasingly choosing to expand the connectivity options available through the LMS, for example by adding personal email, mobile phone and other audio/video communication technologies to facilitate synchronous, especially voice-based, negotiations: To communicate with each other, Tudak [villagers] used Adobe Connect®, the [LMS] forum, email, and we even phoned and texted each other on occasion. We often used more than one Australasian Journal of Educational Technology, 2016, 32(5). 43 communication method simultaneously during meetings. We did try live chat but found it didn’t suit us as much. We successfully used [the LMS] wiki in the first phase of the game. (student feedback, 2011) We responded by adding LMS tools and plugins, such as synchronous audio visual and voice chat media (e.g., Adobe Connect®) from 2011. Students responded positively: “the real-time interaction in this game was very welcome to us external students” (student feedback, 2011). Disappointingly, the LMS tools were somewhat unreliable, but students were quick to adjust: That our “miners” were distributed over a number of time zones … didn't make communication any easier. As a group, Adobe [Connect]® didn’t work … so we resorted to the conventional Skype®. On the last day [of the game], we were all glued to Skype®, simultaneously checking [the LMS]. (student feedback, 2011) Overall, students chose to use a rich mix of synchronous and asynchronous communication technologies, which gave rise to social as well as learning benefits, as these examples of student feedback illustrate: Wiki, discussions on forums and group meetings help to facilitate intra-group communication while … forums and Adobe [Connect]® help to facilitate inter-group communication. I like the way wiki helps us in Phase I since each person can contribute our own part to the group statement and others can view and revise directly, which really saved time. Although we got some troubles with Adobe Connect®, it isn’t a big constraint in inter-group communication … [student] and I used [email] instead for negotiating the contract. (student feedback, 2011) Communication was the key to the game … we utilised the Forums as our main point of discussion and the wiki was where we shared documents for review before they were distributed. We didn't utilise Adobe Connect® many times as we found it difficult with different schedules, time zones etc., but I don't think it was crucial to our interaction although it was great to be able to chat to each other all at once and put a voice to the face and posts! (student feedback, 2011) Pedagogical outcomes: the conversational framework in action As explained previously, Laurillard’s conversational framework describes an effective learner-centred use of technologies as one that ensures “a dialogic process between ‘teacher’ and ‘student’ on two levels, the discursive level, where the focus is theory, concepts, description-building, and the experiential level, where the focus is on practice, activity, procedure-building” (Laurillard, 2007, p. 159), with four potentially concurrent phases: discursive, interactive, adaptive and reflective (Figure 1). After four iterations of RISOS on the LMS platform (2014), when we were sure that the DBR process had led to a stable design, we used the checklist we had derived from Laurillard’s analytical questions (Figure 3) to map the actual dialogic processes occurring between RISOS students and teachers. Overall we found that RISOS did indeed engage students at both levels, through all four stages of the Framework (Figure 5). The discursive stage was evidenced by the students’ engagement with the theory behind the role play through pre-game online lectures and readings, and in-game information sources which were either public or group-only. The public information included relevant game material available to all, such as “media releases” issued regularly in an open forum. From the game’s outset (concurring with advice by Russell & Shepherd, 2010), there was also guidance designed to clarify online etiquette, and to scaffold the initial learning pathways for students potentially anxious about engaging with a learning approach new to them and worth 70% of their grade. For example, students were advised that, just as in real life development contexts, they could expect complexity both in the scenario and in the online environment: This is a dynamic game and all the actors are provided with some basic objectives and information. Beyond this, you need to be creative and improvise to fill in the gaps, or if you're really stuck ask [the lecturer] … Recognise that you won’t understand everything straight away … Some initial confusion is part of the process – but if this persists for too long or becomes uncomfortable contact [the lecturer]. (Extract from Introduction to RISOS) Australasian Journal of Educational Technology, 2016, 32(5). 44 Figure 5. The lived experience of the conversational framework in RISOS 2012–2014 (after Laurillard et al., 2000) We also provided two sets of unique group-only information: a role description that explained the group’s focus and specific objectives and a resource folder, to which new material could be added by the lecturer during the game (equivalent to new information coming to light). As in real life, no group was sure what information other groups had, nor what they wanted to achieve. This approach was a crucial contribution to the authenticity of the learning: This game really forced me to get into character and make decisions based on limited and potentially false information. Due to this I am better able to understand interrelationships as they occur on the ground because I can appreciate how quickly people are misled or feel the need to make hasty decisions. (student feedback, 2011) The framework’s adaptive stage was evident when, for example, the lecturers’ expert knowledge of authentic situations in the country allowed for role play conditions to be adjusted realistically so as to create potential conflict over land amongst the villages. For example, media releases from local government representatives might suggest unexpected political influences or natural disasters that impacted on one or more stakeholders. The interactive stage was also clearly demonstrated: all students and both lecturers engaged in RISOS throughout its 8-week duration. The intensity of lecturers’ input was critical to the students’ buy-in, as they recognised the academics’ own investment in RISOS as a serious learning activity. Again the acknowledged authenticity was crucial: “the ability to ‘live out’ course concepts through the RISOS game was paramount to cementing and retaining the knowledge from this course” (student feedback, 2012). In our analysis, we found that the adaptive and interactive stages cycled through from one to the other multiple times during the most intense part of the role play, as all new information given to one or more groups added another dimension to the interactivity. The framework’s final reflective stage was represented by a comprehensive debriefing. Like Ertmer et al. (2010), we believe that a role play’s effectiveness as a learning strategy depends as much on a thorough analysis of what happened, and why, as on the actual role play activities. While a debrief had thus always been part of the RISOS design, the DBR feedback collected in 2010 and 2011 showed just how critical the debriefing was to students’ understanding of the wider context and the relevance of RISOS to their learning Australasian Journal of Educational Technology, 2016, 32(5). 45 as professionals. This feedback led us to increase the scope and duration of the debriefing from 2012 onwards, and emphasising its importance by making it part of the assessment process. At the game’s conclusion, therefore, students reflected openly on the central learning themes, group learning, and technological/game issues in an online debriefing forum (available for 1 week), then reviewed their personal engagement and learning in relation to key development theory and concepts in an individually written reflective paper. This emphasis on the framework’s reflective stage has helped students bring their learning out of the role play and into the real world, exemplified by this student feedback: “Having participated in the RISOS game, it seems to me now a powerfully accurate model of the forces at play in the field of development” (student reflection paper, 2012). Conclusion In developing an online role play that deals with complex issues, educational designers and teachers run the risk of diminishing authenticity by opting for less complexity. Accepting that an evolutionary process was essential, and building the role play around both a conceptual model (in this case, Laurillard’s conversational framework) and an iterative research paradigm (in this case, DBR), provided us with the capacity to develop and refine a very intense and effective learning experience. Even though the scope of available communication tools in RISOS has been limited essentially to those provided by the university LMS, this has not unduly compromised our capacity to develop a relatively complex learning design for a simple virtual world, with very effective pedagogical outcomes. Exciting as it might be to go deeper into creating authenticity with purpose-made audiovisual material, even perhaps investing in an immersive visual, virtual RISOS environment, we believe that making the artifice more obvious might risk more behavioural stereotyping, thus diminishing authentic learning. Among many other outcomes, our ongoing monitoring of what students were doing and why – facilitated by our DBR methodology and effective debriefing – showed that they were constantly seeking better ways of connecting with one another and choosing their own communication technologies to support those connections. As we could not match the speed of external innovation within the institutional LMS, we chose to embrace the dialogic pathways opened up by the students’ choices to add new technologies into the mix of RISOS communication tools, consistent with our focus on the conversational framework (Laurillard, 2002). However, this did create challenges in terms of assessing engagement and group work: the greater the diversity of technologies being used, and the fewer text records, the more rigorous and criterion-referenced must be the approach to assessing the contributions and leadership evidenced by individual students. Nevertheless, we count it as a measure of the success of RISOS as a professional learning activity that the students wanted to mimic their usual communication approaches to accomplish their work. We believe awareness of this aspect of students’ behaviour in online role plays is of great relevance to designers of online education, as it illustrates how today’s students mark their ownership not only of their learning activities but also of the technologies used in those activities. In summary, in developing the online role play RISOS, we have created a minimalist virtual world in which postgraduate students enjoy an immersive experience as they role play key actors on a land rights stage. In this way, we have been able to facilitate personal, deep and socialised learning experiences focused on the complex issues of consultation, negotiation and decision-making, available to students regardless of their geographic location. Acknowledgements The authors acknowledge the contribution to RISOS of co-lecturer Dr Colin Filer, and technological specialists Megan Poore (2009) and Jennifer Jones (2010). We also appreciate time spent at the Australian National University Academic Women’s Writing Workshops 2012 and 2014, and thank the relevant funding bodies. Australasian Journal of Educational Technology, 2016, 32(5). 46 References Anderson, T., & Shattuck. J. (2012). Design-based research. A decade of progress in education research? Educational Researcher, 41(1), 16–25. http://dx.doi.org/10.3102/0013189X11428813 Bannan-Ritland, B. (2003). The role of design in research: the integrative learning design framework. Educational Researcher, 32(1), 21–24. http://dx.doi.org/10.3102/0013189X032001021 Bateson, G. (1972). Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution and epistemology. San Francisco, CA: Chandler. Beckmann, E. A. (2010). Learners on the move: Mobile modalities in development studies. Distance Education, 31(2), 159–173. http://dx.doi.org/10.1080/01587919.2010.498081 Beckmann, E. A. & Kilby, P. (2010). Transnational postgraduate study for development workers: Using technology to bridge the gap. In S. Mukerji. & P. Tripathi (Eds), Cases on technological adaptability and transnational learning: Issues & challenges (pp. 102–122). Hershey: IGI Global. http://dx.doi.org/10.4018/978-1-61520-779-4.ch006 Bessant, J., & Francis, D. (1999). Developing strategic continuous improvement capability. International Journal of Operations & Production Management, 19(11), 1106–1119. http://dx.doi.org/10.1108/01443579910291032 Biggs, J., & Tang, C. (2011). Teaching for quality learning at university. Maidenhead: Open University Press & McGraw-Hill. Cornelius, S., Gordon, C., & Harris, M. (2011). Role engagement and anonymity in synchronous online role play. International Review of Research in Open and Distance Learning, 12(5), 57–73. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/923/1857 Dede, C., Ketelhut, D. J., Whitehouse, P., Breit, L., & McCloskey, E. M. (2009). A research agenda for online teacher professional development. Journal of Teacher Education, 60(1), 8–19. http://dx.doi.org/10.1177/0022487108327554 Design Based Research Collective. (2003). Design based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. http://dx.doi.org/10.3102/0013189X032001005 Ellington, H. (I.) (2004). Using games, simulations, case studies and role-play to stimulate students’ creativity. Science Activities, Classroom Projects and Curriculum Ideas, 49(3), 71–76. Retrieved from http://dx.doi.org/10.1080/00368121.2011.632449 Ertmer, P. A., Strobel, J., Cheng, X., Chen, X., Kim, H., Olesova, L., Sadaf, A., & Tomory, A. (2010). Expressions of critical thinking in role-playing simulations: Comparisons across roles. Journal of Computing in Higher Education, 22(2), 73–94. http://dx.doi.org/10.1007/s12528-010-9030-7 Freeman, M. A., & Capper, J. M. (1999). Exploiting the web for education: An anonymous asynchronous role simulation. Australian Journal of Educational Technology, 15(1), 95–116. http://dx.doi.org/10.14742/ajet.1849 Gredler, M. (1992). Designing and evaluating games and simulations: A process approach. London: Kogan Page. Harris, M., Cornelius, C., & Gordon, C. (2009). On-line learning, role-play and reflection to encourage professional insight through knowledge transfer on a work-based professional educator programme. Work-based Learning e-Journal. Retrieved from http://www.wblearning- ejournal.com/peerreview11.pdf Herrington, J., Reeves, T. C., & Oliver, R. (2010). A guide to authentic e-learning. New York, NY: Routledge. 
 Ip, A., Linser, R., & Naidu, S. (2001, April). Simulated worlds: Rapid generation of web-based role-play. Paper presented at AusWeb01, 7th Australian World Wide Web Conference, Coffs Harbour. Retrieved from http://ausweb.scu.edu.au/aw01/papers/refereed/ip/ Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies. London: RoutledgeFalmer. Laurillard, D. (2007). Pedagogical forms for mobile learning: framing research questions. In N. Pachler (Ed.), Mobile learning: Towards a research agenda (pp. 153–175). London: WLE Centre, Institute of Education, University of London. Retrieved from http://eprints.ioe.ac.uk/627/1/Mobile_C6_Laurillard.pdf Laurillard, D. (2009). The pedagogical challenges to collaborative technologies. International Journal of Computer-Supported Collaborative Learning, 4(1), 5–20. http://dx.doi.org/10.1007/s11412-008-9056- 2 http://dx.doi.org/10.3102/0013189X11428813 http://dx.doi.org/10.3102/0013189X032001021 http://dx.doi.org/10.1080/01587919.2010.498081 http://dx.doi.org/10.4018/978-1-61520-779-4.ch006 http://dx.doi.org/10.1108/01443579910291032 http://www.irrodl.org/index.php/irrodl/article/view/923/1857 http://dx.doi.org/10.1177/0022487108327554 http://dx.doi.org/10.3102/0013189X032001005 http://dx.doi.org/10.1080/00368121.2011.632449 http://dx.doi.org/10.1007/s12528-010-9030-7 http://dx.doi.org/10.14742/ajet.1849 http://www.wblearning-ejournal.com/peerreview11.pdf http://www.wblearning-ejournal.com/peerreview11.pdf http://ausweb.scu.edu.au/aw01/papers/refereed/ip/ http://eprints.ioe.ac.uk/627/1/Mobile_C6_Laurillard.pdf http://dx.doi.org/10.1007/s11412-008-9056-2 http://dx.doi.org/10.1007/s11412-008-9056-2 Australasian Journal of Educational Technology, 2016, 32(5). 47 Laurillard, D., Stratfold, M., Luckin, R., Plowman, L., & Taylor, J. (2000). Affordances for learning in a non-linear narrative medium. Journal of Interactive Media in Education, Issue 2, article 2. http://dx.doi.org/10.5334/2000-2 Maier, H. R., Baron, J., & McLaughlan R. G. (2007). Using online roleplay simulations for teaching sustainability principles to engineering students. International Journal of Engineering Education, 23(6), 1162–1171. Retrieved from http://www.ingentaconnect.com/content/intjee/ijee/2007/00000023/00000006/art00013 McLaughlan, R., & Kirkpatrick, D. (2014). Peer learning using computer supported roleplay-simulations. In D. Boud, R. Cohen, & J. Sampson (Eds.), Peer learning in higher education: Learning from and with each other (pp. 141–155). London: Kogan-Page. Naidu, S., Ip, A. & Linser, R. (2000). Dynamic goal-based role-play simulation on the web: A case study. Educational Technology & Society, 3(3), 190–202. Retrieved from http://www.ifets.info/journals/3_3/b05.html Reeves, T., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 97–116. http://dx.doi.org/10.1007/BF02961476 Russell, C., & Shepherd, J. (2010). Online role-play environments for higher education. British Journal of Educational Technology, 41(6), 992–1002. http://dx.doi.org/10.1111/j.1467-8535.2009.01048.x Schank, R. C., Fano, A., Bell, B., & Jona, M. (1994). The design of goal-based scenarios. Journal of the Learning Sciences, 3(4), 305–345. http://dx.doi.org/10.1207/s15327809jls0304_2 Sullivan, P. (2002). “It’s easier to be yourself when you are invisible”: Female college students discuss their online classroom experiences. Innovative Higher Education, 27(2), 129–144. http://dx.doi.org/10.1023/A:1021109410893 Teahan, J. E. (1975). Role-playing and group experiences to facilitate attitude and value changes among black and white police officers. Journal of Social Issues, 31(1), 35–45. http://dx.doi.org/10.1111/j.1540-4560.1975.tb00738.x Wang, F., & Hannafin, M. J. (2005). Design based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23. http://dx.doi.org/10.1007/BF02504682 Westera, W. (2011). On the changing nature of learning context: Anticipating the virtual extensions of the world. Educational Technology & Society, 14(2), 201–212. Retrieved from http://www.ifets.info/journals/14_2/17.pdf Wills, S., Leigh, E., & Ip, A. (2011). The power of role-based e-learning: Designing and moderating online role play. Oxford: Routledge. Wills, S., Rosser, E., Devonshire, E., Leigh, E., Russell, C., & Shepherd, J. (2009). Encouraging role based online learning environments by building, linking, understanding and extending: The BLUE report. Surry Hills: Australian Learning and Teaching Council. Retrieved from http://ro.uow.edu.au/asdpapers/116/ Corresponding author: Elizabeth Beckmann, elizabeth.beckmann@anu.edu.au Australasian Journal of Educational Technology © 2016. Please cite as: Beckmann, E. A., & Mahanty, S. (2016). The evolution and evaluation of an online role play through design-based research. Australasian Journal of Educational Technology, 32(5), 35-47. http://dx.doi.org/10.14742/ajet.1957 http://www.ingentaconnect.com/content/intjee/ijee/2007/00000023/00000006/art00013 http://www.ifets.info/journals/3_3/b05.html http://dx.doi.org/10.1007/BF02961476 http://dx.doi.org/10.1111/j.1467-8535.2009.01048.x http://dx.doi.org/10.1207/s15327809jls0304_2 http://dx.doi.org/10.1023/A:1021109410893 http://dx.doi.org/10.1111/j.1540-4560.1975.tb00738.x http://dx.doi.org/10.1007/BF02504682 http://www.ifets.info/journals/14_2/17.pdf http://ro.uow.edu.au/asdpapers/116/ mailto:elizabeth.beckmann@anu.edu.au http://dx.doi.org/10.14742/ajet.1957 The evolution and evaluation of an online role play through design-based research Introduction A role play is an activity that requires students to assume a specific identity (either a named historical character or a character designated by occupation, nationality, culture, worldview, or any combination of these traits); research the views/expe... Role plays are particularly valuable as intensive opportunities for interactive engagement and learning, especially among professionals (Harris, Cornelius, & Gordon, 2009; Maier, Baron, & McLaughlan, 2007). Ertmer et al. (2010) argue that, in professi... While many researchers distinguish between role plays, simulations, and games (e.g., Ellington, 2004; Russell & Shepherd, 2010), RISOS has elements of all three. It has specific roles that students must adopt; it has the “tactical decision” (essential... Over 6 years, RISOS evolved from a 2-hour, face-to-face, non-assessed course activity available only to on-campus students into a substantive, intellectually demanding 8-week online role play that contributes 70% to the course grade. RISOS allowed on-... Results and discussion In summary, in developing the online role play RISOS, we have created a minimalist virtual world in which postgraduate students enjoy an immersive experience as they role play key actors on a land rights stage. In this way, we have been able to facili... Acknowledgements The authors acknowledge the contribution to RISOS of co-lecturer Dr Colin Filer, and technological specialists Megan Poore (2009) and Jennifer Jones (2010). We also appreciate time spent at the Australian National University Academic Women’s Writing W... References