Microsoft Word - [5] 2450-9245-2-LE Australasian Journal of Educational Technology, 2016, 32(2). 64 ascilite Design and evaluation of an online tool for open learning with blogs Hans Põldoja Tallinn University, Estonia Erik Duval Katholieke Universiteit Leuven, Belgium Teemu Leinonen Aalto University, Finland Blogs are used in higher education to support face-to-face courses, to organise online courses, and to open up courses for a wider group of participants. However the open and distributed nature of blogs creates problems that are not common in other learning contexts. Four key challenges related to the use of blogs in learning were identified from earlier research: fragmented discussions, a lack of coordination structures, weak support for awareness, and a danger of over-scripting. The EduFeedr system has been designed to address these issues. In this paper, the authors present their evaluation of its design and effectiveness in a total of 10 courses. The results indicate that learners find the EduFeedr system useful in following discussions and in comparing their progress with other learners. The coordination and awareness issues are seen as more important than the fragmentation of discussions and a danger of over-scripting. Introduction Blogs are used in higher education to provide a space for reflection, a forum for discussions, a portfolio of completed assignments, and for opening up courses for a wider group of participants. While some recent research has focused on the pedagogical aspects of using blogs in higher education, Sim and Hew (2010) suggest that one focus of future research should be the development of web technologies that will enhance the conversational and interactive aspects of blogging. Our study focuses on designing and evaluating an online tool that aims to address some of the issues that impede the use of blogs in online and blended learning courses. A blog is a website where the content is comprised of posts that are displayed in reverse chronological order. A typical blog is a personal website that is written by a single person; however it is also possible to have several authors. Readers can become engaged by writing comments on blog posts. Syndication technologies such as really simple syndication (RSS) and Atom enable readers to receive new posts and comments automatically. All blogs and their interconnections are often referred to as the blogosphere. The blogosphere can be seen both as a social network and as an ecosystem. The possibilities for using blogs in learning became evident soon after blogs emerged (Oravec, 2003; Williams & Jacobs, 2004). Sim and Hew (2010) identified six major applications for blogs in education: (a) maintaining a learning journal, (b) recording personal life, (c) expressing emotions, (d) communicating with others, (e) assessment, and (f) managing tasks. Kim (2008) suggests that the use of blogs may help to overcome various limitations of other computer- mediated communication systems, such as difficulties in managing communication, passiveness of students, lack of ownership, instructor-centeredness, and limited archives of communication. Previous studies show that reading other blogs and receiving feedback on one’s own blog posts were the more effective aspects of using blogs in learning (Churchill, 2009; Ellison & Wu, 2008). Blogs are useful in disciplines that require students to discuss, write, reflect, and make comments about content or ideas (Cakir, 2013). Blogging has been found particularly beneficial in teacher education because it can motivate learners, foster collaboration and cooperation, promote different instructional practices, and enrich the learning environment (Goktas & Demirel, 2012). Teachers who acquire these competences during the blogging assignments can later apply these methods in their own teaching. Australasian Journal of Educational Technology, 2016, 32(2). 65 ascilite Blogs are a common platform for creating a personal learning environment (PLE). Dabbagh and Kitsantas (2012) distinguish between three levels of blog use for self-regulated learning in PLEs: (a) using blogs for personal information management, (b) using blogs for social interaction and collaboration, and (c) using blogs for information aggregation and management. In the first level, learners use a blog as a private journal to set learning goals and plan for course assignments. In the second level, they make blogs public and allow others to comment on their posts. In the third level, learners integrate their blogs with other web services, for example by adding the blog to an RSS aggregation service. These three levels are related to the six major educational uses of blogs referred to earlier in this introduction (Sim & Hew, 2010): personal information management involves keeping a learning journal and recording personal life, social interaction and collaboration relates to communication with others, while information aggregation and management can be associated with managing tasks and assessment. This research paper partly addresses all three aspects of using blogs as PLEs, but focuses primarily on the third level by studying the aggregation and management tools for online courses. One of the important characteristics of blogs is openness. Blogs follow the basic principles of the open web such as decentralisation and universality (Berners-Lee, 2010). The decentralised architecture of blogs allows anybody to set up a blog on their own server or use a free blog hosting site. Universality enables any blog post or page to be linked to by using a unique address. The open nature of blogs makes blogging especially suitable for pedagogical approaches that emphasise public discourse, open participation and self-directed learning (McLoughlin & Lee, 2007). In these types of learning contexts, it is common for learners to be connected to more than one educational institution. The authors’ initial motivation to study the coordination of blog use in online courses evolved from a number of courses in which blogs were used to open up the course for external participants. Open online courses with blogs The term open learning refers to learning that takes place online and is more or less open for anyone to participate in. In this way, open learning is an umbrella term that covers various kinds of online courses, such as massive open online courses (MOOCs) and blog-based open online courses. This paper however, does not address classical open distance learning, but focuses specifically on using the open web for learning. A MOOC is an online course with open access and a large number of participants. Some MOOCs use special platforms such as Coursera, Udacity or edX. These courses are often called x-MOOCs (Rodriguez, 2013). Other type of MOOCs – known as c-MOOCs – are loosely structured and allow learners to use a range of different online tools, such as blogs, Twitter, social bookmarking, feed readers and web conferencing (Fini, 2009). This study was carried out in a context where formal higher education courses were opened up for external participants. These courses differentiated from typical c-MOOCs by having a more predefined structure and a smaller number of participants. Both the university students and the external participants used their personal blogs to submit course assignment and to participate in discussions. The moderate number of participants (under 50) allowed the course facilitator to follow and interact with every learner. Due to the smaller number of participants, a smaller variety of online tools was used. The authors propose however, that some of the results can be applied in all types of online courses with blogs. Critical issues in blog-based courses Earlier research identified several critical issues in the use of blogs in learning. These have been identified as fragmented discussions, the lack of coordination structures, weak support for awareness, and the danger of over-scripting. The conversation used in blogs differs from online forums and mailing lists in several ways. Firstly, there is a problem with the distributed and fragmented nature of blog discussions (Efimova & de Moor, 2005) that may slow down the pace of conversation. Responses may be posted as comments to the original post or as separate posts in another blog. Thus a discussion can take place in an ecosystem of blogs rather than in a single blog. In forum discussions, people will easily notice new or active threads. In a distributed group of blogs, people may only visit or subscribe to a selection of blogs. Sometimes an interesting Australasian Journal of Educational Technology, 2016, 32(2). 66 ascilite conversation topic is noticed only when other bloggers link to it. Xie, Ke, and Sharma (2008) concluded that the lack of a central location for all blogs prevents students from learning from the blogging activities of other course participants. In a course setting where participants prefer different communication tools, conversations started in blogs may move to other communication channels. Expressed in a different way by Kop, Fournier, and Mak (2011), “too much freedom in choice of tools unnecessarily fragments the conversation unless other tools are used to recombine the process” (p. 86). Secondly, there is a problem with the lack of structure for coordination, also called articulation, that is the process of managing dependencies among activities (Malone & Crowston, 1994). Cooperative work that involves multiple actors requires that activities are coordinated, scheduled, aligned, meshed, integrated, etc. (Schmidt & Simone, 1996). Enrolling in the course, managing assignments and getting an overview of learner progress are examples of articulation that require coordination mechanisms. Thirdly, coordination is related to awareness support mechanisms, typically implemented as a notification system that provides information about the presence, tasks, and actions of collaborators. Social awareness includes online presence and connections between actors, action awareness focuses on individual actions, and activity awareness deals with long-term joint endeavours directed at major goals (Carroll, Neale, Isenhour, Rosson, & McCrickard, 2003). The genre of blog-based courses includes the scenario of a teacher writing assignment specifications to his/her blog and students writing responses to these assignments in their blogs. The students are expected to participate in collaborative knowledge building by commenting on and linking to the blog posts of each others. This requires the teacher to plan the tasks and the expected collaboration in the course. For instance, scripting can be used to coordinate these learning activities (O’Donnell & Dansereau, 1992). A script defines how students should form groups, how they should interact and collaborate, and how they should do the assignments. The main components of pedagogical scripts are: (a) task definition, (b) group definition, (c) distribution, (d) mode of interaction, and (e) timing (Dillenbourg, 2002). A coordination mechanism should support the implementation of scripts in online courses. However, too rigid coordination may result in over-scripting the course and in turn may disturb natural interactions and problem solving processes, increase the cognitive load of learners, force teacher-controlled learning processes, and impede learners from establishing and achieving their own learning goals (Dillenbourg, 2002). Regardless of these challenges, the positive aspects and the potential benefits of using blogs in online courses are remarkable. Open learning with blogs supports learner-centeredness, active participation of students, and increased ownership of personal and group learning. While blogs are by nature a learner- centered medium in education, it is important to look also at the teachers’ perspective. Some of the recent research emphasises the importance of awareness tools for teachers. For instance, Dillenbourg et al. (2011) suggest that the design of learning tools should not focus only on individual learners and groups, but also on usability at the classroom or course level where teachers have to orchestrate multiple learning activities within a variety of real life constraints. Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, and Dimitriadis (2014) point out that teachers need awareness tools to adjust their pedagogical scripts according to the progress of the learning process. Research questions and the structure of the paper Our study focuses on the teachers’ perspective in open learning with blogs. This paper addresses the following four interlinked research questions (RQ 1–4) and design challenges. How and to what extent can an online tool: 1. combine and present the fragmented discussions in blog-based courses? 2. provide coordination structures for managing blog-based courses? 3. provide a notification system for supporting awareness of learners and facilitators? 4. prevent over-scripting while providing sufficient coordination? The rest of the article is divided into five sections. The second section introduces the design and implementation of EduFeedr. The third section provides an overview of the research methodology. This Australasian Journal of Educational Technology, 2016, 32(2). 67 ascilite is followed by the analysis of the results and discussion of the findings. The article ends with the conclusions drawn from the study. Design and implementation of EduFeedr The design process In order to address the research questions and design challenges summarised in the previous section, the authors have designed and developed a prototype of an online learning tool referred to as EduFeedr. The design process of EduFeedr follows a research-based design approach that is based on four iterative stages: (a) contextual inquiry, (b) participatory design, (c) product design, and (d) the development of software prototype as hypothesis, where the prototype aims to answer certain challenges recognised in earlier phases of research (Leinonen, Toikkanen, & Silfvast, 2008). In the contextual inquiry phase, 4 blog-based courses were organised in which the authors tested various approaches for coordinating the learning activities (wikis, generic feed readers, etc.). The use of multiple coordination tools that were not directly integrated introduced a considerable amount of additional work for the facilitator. This experience helped to define the design challenges and requirements for coordinating blog-based courses. Based on the research questions, the following initial design requirements were decided: (a) the system should aggregate information from blogging platforms using open standards (RSS, Atom); (b) the system should not require any special plug-in in student blogs; (c) the scope of the system should be limited to aggregating and annotating the feeds from the blogs and visualising the process of knowledge building; (d) only the facilitator should be required to have a user account in the system to modify course settings; and (e) anyone should have read access to the aggregated course content. Design requirement (a) is related to combining the fragmented discussions (RQ 1). Requirement (c) is linked both to providing coordination structures (RQ 2) and supporting awareness (RQ 3). Requirements (b), (d) and (e) are general design requirements in order to keep the architecture of the system open and lightweight. In the participatory design phase, the scenario-based design method (Carroll, 2000) was used to gather user feedback for the initial design ideas. A scenario-based design approach requires the designers to focus on people’s activities and envision how these activities could be supported by software. The authors wrote six short scenarios regarding the use of EduFeedr: (a) first experience with EduFeedr, (b) posting a response to the assignment in the student blog, (c) exploring the connections between student blogs, (d) setting up course feeds, (e) archiving course posts and comments, and (f) using the offline client. An example scenario about exploring the connections between the blogs is presented in Figure 1. John has been using EduFeedr for a few weeks. For him, the most exiting feature is a way of presenting connections between the blogs. EduFeedr has a visualisation in which all the blogs are displayed as nodes. Lines between the nodes show the links between the blog posts. All the students have linked to the course blog. Some of the student blogs have a lot of connections while others have not been so active. It is possible to switch on a different view and see who has commented on which blog. This time, John finds out that some student blogs have actually received more comments than his blog. The same information is also displayed as a table in which it is easy to see how many pingbacks and comments each participant has made. EduFeedr has also aggregated all the comments. It means that John can see all the comments that one particular student has made on a same page without visiting all the blogs. This will save him a lot of time because commenting is part of his grading scheme and students get points for that. Figure 1. Example scenario: Exploring the connections between student blogs A list of questions was compiled for each scenario in order to guide the discussion. The scenarios were evaluated in two design sessions with 5 university teachers who had used blogs in their courses. In the design session, each participant received a sheet of paper with a written scenario. After the participants Australasian Journal of Educational Technology, 2016, 32(2). 68 ascilite had read the scenario, the designer facilitated a discussion based on the guiding questions. The process was repeated for all six scenarios. In the product design phase, the authors defined use cases and basic interaction by writing user stories (Cohn, 2004) and creating paper prototypes (Snyder, 2003). User stories capture a software requirement from the end user perspective in one or few sentences. In total, 48 user stories were written. The design team evaluated the user stories and paper prototypes internally. Two teachers were involved in discussing the paper prototypes. Developing user stories and paper prototypes in parallel helped to identify missing functionalities and to clarify details. Some initial paper prototypes are presented in Figure 2. Figure 2. Initial paper prototypes of the course front page The final phase of research-based design is the development of a software prototype as a hypothesis. In this study, the hypothesis is that the coordination of blog-based courses and participants’ awareness could be improved by using a web-based tool that aggregates, structures and presents the learning activities that take place in learners’ personal learning environments. The software prototype that aims to address the challenges is the EduFeedr, a web-based tool for open learning with blogs. Implementation of EduFeedr EduFeedr enables anyone to view the aggregated information from the courses. Facilitators need a user account only for managing the course settings and accessing contact information of learners. Each course is divided into six sections: (a) course feed, (b) course information, (c) participants, (d) assignments, (e) progress, and (f) social network. The course feed page combines the fragmented discussions by presenting recent posts and comments from the participants’ blogs (see Figure 3). Australasian Journal of Educational Technology, 2016, 32(2). 69 ascilite Figure 3. User interface of EduFeedr: Course feed page (left) and progress page (right) EduFeedr provides coordination structures for managing the course participants and assignments. Learners can enrol in the course by submitting the address of their blog. The list of participants can be exported in various formats: vCard for address book, outline processor markup language (OPML) for feed readers, and spreadsheet for grading the assignments. Assignment specifications are written and published by the facilitator as blog posts in the course blog. Blog posts containing assignment specifications have to be specified in EduFeedr together with a due date. If a learner’s post includes a link to the assignment specification in the course blog, the system will recognise it as a submission to the assignment. As a backup strategy, the system relies on the posting date to connect assignments and submissions. In order to raise participants’ awareness, EduFeedr provides a progress chart that displays submitted assignments (see Figure 3) and a social network visualisation that connects learners who have commented on or linked each other’s posts. The social network data can be exported in tab-separated format for using in other visualisation tools. EduFeedr leaves a reasonable amount of flexibility in scripting the learning activities. Tasks are defined in the facilitator’s blog posts. Learners may choose their personal tools to form smaller groups, distribute work and interact with their peers. Only the final outcome must be submitted as a blog post. Learners may also use shared group blogs instead of personal blogs in EduFeedr. This flexibility allows users to vary and improvise their plans during the course. Table 1 summarises how the critical issues of blog-based courses were addressed in EduFeedr. Table 1 Addressing the critical issues of blog-based courses in EduFeedr Issue Addressing the issue in EduFeedr Fragmented discussions Course feed page displays recent blog posts and comments The lack of coordination structures Learners can self-enroll in the course Learners can add their blog to EduFeedr for aggregation Facilitators can manage course participants Facilitators can manage assignments Weak support for awareness Progress page displays learners’ submissions for assignments Social network page displays social network of learners The danger of over-scripting Facilitators post blogs as pedagogical scripts to which the learners are expected to respond in their own blog in their own way Learners have the flexibility to use other social media tools in addition to blogs and EduFeedr Australasian Journal of Educational Technology, 2016, 32(2). 70 ascilite Technically, EduFeedr is developed as a plug-in for the Elgg social networking engine. The Elgg framework provided the developers with user management and plugin architecture. EduFeedr operates as a front end that displays aggregated course data from the local database. Aggregation of blog posts and comments is handled by a separate web service. Various open-source libraries such as SimplePie, JSViz, and NuSOAP have been used. The source code is available under the GNU General Public License. Related systems There are a number of similar systems for coordinating blog-based learning environments. gRSShopper is aggregator and newsletter software that has been used in several c-MOOCs (Downes, 2010). The system aggregates all the blog posts that contain the course tag and enables the facilitator to compile a daily newsletter based on selected blog posts and Twitter tweets. eMUSE is a learning environment that can aggregate content from various social media platforms such as Blogger, Delicious, Twitter, YouTube, SlideShare, MediaWiki, and others (Popescu, 2014). The instructor can view different visualisations showing student progress and grade their assignments. StepUp! has also focus on visualisations that display the number of social media artefacts (blog posts, comments and tweets) and the time spent on various learning activities (Santos, Verbert, Govaerts, & Duval, 2013). There have been attempts to add blog aggregation to learning management systems, such as the BIM blog aggregation module for Moodle (Jones, 2013). Relying only on RSS feeds limits the communication between the blogs and the course management tool. LePress (Tomberg, Laanpere, Ley, & Normak, 2013) and Dippler (Laanpere, Pata, Normak, & Põldoja, 2014) have addressed this issue by developing a special plug-in for learners using WordPress blogs. This enables two-way communication that may include privately submitted assignments, private feedback, and grading. On the other hand, architectures that require a special plug-in in learners’ blogs are not suitable for open online courses in which learners typically use free blog hosting services that do not allow the installation of additional plug-ins. In addition to syndication feeds, eMUSE and StepUp! use application programming interfaces (API) of Twitter and other systems to aggregate additional content. Table 2 presents the comparison of related systems, their aggregation architecture, supported blogging services, and coordination features. Table 2 Comparison of related systems System Aggregation architecture Supported services Coordination features EduFeedr Syndication feeds WordPress, Blogger Courses, assignments, visualisations gRSShopper Syndication feeds Any blogging platform Newsletters, archive eMUSE Syndication feeds, service APIs Blogger, MediaWiki, Delicious, Twitter, YouTube, Picasa, SlideShare Courses, visualisations, grading BIM Syndication feeds, plugin for Moodle Any blogging platform Assignments, grading StepUp! Syndication feeds, service APIs Any blogging platform, Twitter, Toggl Courses, visualisations, time tracking, Open Badges LePress Plugin for WordPress WordPress Courses, assignments, grading Dippler Plugin for WordPress WordPress Courses, assignments, learning resources, competences, grading Among these systems, gRSShopper is the closest to EduFeedr, since both systems are designed for supporting open online courses. However, gRSShopper does not support the management of assignments and lacks visualisations of learning activities. Currently, the EduFeedr system focuses only on Australasian Journal of Educational Technology, 2016, 32(2). 71 ascilite aggregating blog posts and comments, while some other systems aggregate other types of content as well. Some of the visualisations available from eMUSE and StepUp! could serve as examples for developing additional visualisations for EduFeedr. Methods In the design phase, this study employed a number of interaction design methods such as scenario-based design, user stories, and paper prototyping. An agile software development methodology was followed, in which the software is divided into incremental functional versions that can be released frequently. This allowed the system to be tested in real courses with real participants and to gather both quantitative and qualitative data to guide the design process and the evaluation study. This paper presents the results from the use of two instruments to evaluate the latest version of EduFeedr: (a) a survey with learners and facilitators who have used the system recently, and (b) an analysis of usage based on Google Analytics. The survey method was chosen because the authors wanted to collect data from users in other institutions and countries in addition to the feedback from learners in their own courses. The survey data and the Google Analytics data were used in conjunction with each other to corroborate interpretations made from them. In practice, the data from Google Analytics has also guided the design of the survey. Participants It was decided to forward the survey to learners who had participated in a course through EduFeedr during the previous year (N = 173) and facilitators who had run courses in EduFeedr during 2 previous years (N = 13). A total of 61 responses were received from learners and 8 responses from facilitators. Of these, 3 learners and 1 facilitator completed the survey only partially, but the incomplete data gathered from them was also used when possible. The learners who responded had participated in 10 different courses. The majority of the respondents had participated in master level courses (N = 39), but there were also 11 bachelor students, 4 respondents from an in-service teacher-training course, and 5 respondents from an open online course. 60% of the learners who responded were female and 40% male. The average age of the learners was 33.5 years (SD = 10.9). Of the facilitators, 5 were female and 2 were male. The average age of the facilitators was 39.3 years (SD = 6.8). A total of 77% of the learners had used EduFeedr within the last 6 months. During the course, 68% of the learners visited EduFeedr at least once each week. A total of 60% of the learners had participated in only one course in EduFeedr, 8% had taken two courses and 32% had experience from three or more courses. The majority of the respondents also had experience using other online learning platforms. A total of 93% of the learners had experience with learning management systems, 58% were using Moodle at least on a monthly basis. Survey instrument and data collection A web-based survey was divided into four sections: (a) critical issues and important learning activities in blog-based courses, (b) support for these learning activities in EduFeedr, (c) technology acceptance of EduFeedr, and (d) background information about the respondents. In the first three sections, the respondents were required to rate a number of statements on a 7-point Likert scale. Based on the literature review and courses carried out using EduFeedr, the authors identified 16 important learning activities in blog-based courses. The relationships between critical issues and learning activities in blog-based courses are presented in Figure 4. Some of the learning activities are clearly related to one of the issues while others are related to several issues. The survey contained a different set of activities for learners and for facilitators. Activities related to following the course discussions (A1–A4) were the same for all respondents. Learners’ survey included activities related to awareness (A5–A10), while the facilitators had activities related to managing the course (A11–A16). The background information section included various types of questions related to demography, the frequency of using EduFeedr and previous experience with online learning. Considering the differences in the learners’ and facilitators’ questionnaire, the reliability was calculated separately for both survey instruments. The Cronbach’s alpha for the learners’ survey indicated a high internal consistency (α = 0.908). Due to the smaller number of respondents, the facilitators’ survey had an acceptable internal consistency (α = 0.718). Australasian Journal of Educational Technology, 2016, 32(2). 72 ascilite Figure 4. The relationships between critical issues and learning activities Analysis The means (Ms) and standard deviations (SDs) were calculated for each Likert scale statement for learners, facilitators, and all respondents. In order to depict the variation of answers, quartiles were calculated and box plot charts were created showing the sample minimum, lower quartile, median, upper quartile, and sample maximum. Results Critical issues in blog-based courses Critical issues in blog-based courses were rated from not critical at all (1) to very critical (7) (see Figure 5). The most critical issues were the fragmentation of discussions (M = 4.45; SD = 1.61) and the Australasian Journal of Educational Technology, 2016, 32(2). 73 ascilite awareness of learning activities (M = 4.16; SD = 1.84). All four issues were found to be more critical by the facilitators than by the learners. The most noticeable difference between learner and facilitator answers was related to I2. The lack of coordination tools was seen to be a more critical issue by the facilitators (M = 4.88; SD = 1.81) than by the learners (M = 3.79; SD = 1.84). Figure 5. Critical issues in blog-based courses Importance of learning activities in blog-based courses The importance of common learning activities in blog-based courses was rated from not important at all (1) to very important (7) (see Figure 6). Activities related to course assignments were rated as the most important. For learners, the two most important activities were “Being aware of assignments and deadlines” (M = 6.62; SD = 0.69) and “Getting feedback for assignments that I have submitted” (M = 6.59; SD = 0.76). Also, the facilitators rated “Keeping track of submitted assignments” as the most important activity (M = 6.25; SD = 1.04). This can be explained by the fact that most of the respondents were from formal higher education courses. The least important activities for learners were “Understanding my position in the social network of course participants” (M = 4.21; SD = 1.75) and “Following recent comments to other blogs” (M = 4.69; SD = 1.63). The facilitators on the other hand found “Following recent comments to other blogs” to be more important activity (M = 5.63; SD = 0.74). For facilitators, the less important activities were “Saving course data for further analysis” (M = 5.25; SD = 1.16), “Protecting the privacy of course participants” (M = 5.25; SD = 1.39), and “Using my personal tools to follow the course” (M = 5.13; SD = 1.73). Support for important learning activities in EduFeedr The respondents also rated EduFeedr on how it supports the learning activities listed in Figure 6. The statements regarding how EduFeedr supports important learning activities were rated from strongly disagree (1) to strongly agree (7). Both the facilitators and learners agreed that EduFeedr enables them to follow recent blog posts (M = 5.66; SD = 1.57) and comments to other blogs (M = 5.15; SD = 1.68). Learners also tend to agree that EduFeedr helps them to be aware of assignments and deadlines (M = 5.25; SD = 1.58) and to compare their progress with other learners (M = 5.08; SD = 1.69). The facilitators agreed that EduFeedr enables them to manage the list of participants (M = 5.86; SD = 0.69) and to keep track of submitted assignments (M = 5.57; SD = 1.13). The facilitators tend to agree less that EduFeedr makes it easier to grade submitted assignments (M = 4.14; SD = 1.57) and to identify learners who need support (M = 4.29; SD = 1.50). Both the learners and the facilitators tend to disagree that EduFeedr enables them to protect their privacy (M = 3.64; SD = 1.39) or the privacy of course participants (M = 3.57; SD = 1.51). We recommend the participants to use an alias or protect their blog posts with a password if they do not feel comfortable with open blogging. The facilitators also tend to disagree that EduFeedr facilitates saving course data for further analysis (M = 3.86; SD = 0.90). Currently it is possible to export the social network data. Both the facilitators and the learners only somewhat agreed that EduFeedr enables them to use personal tools for following the course (M = 4.43, SD = 1.31). It is possible to download an OPML file that can be used to subscribe to all participants’ blogs. One of the respondents requested that EduFeedr should provide a combined RSS feed that includes all posts in the course. Australasian Journal of Educational Technology, 2016, 32(2). 74 ascilite Figure 6. Learning activities and their support in EduFeedr Technology acceptance of EduFeedr The acceptance of EduFeedr based on the extended technology acceptance model (TAM2) (Venkatesh & Davis, 2000) was also studied. Only sections relevant to EduFeedr were chosen from the TAM2 model (perceived usefulness, perceived ease of use, output quality, result demonstrability). TAM2 statements were rated from strongly disagree (1) to strongly agree (7), as recommended by Venkatesh and Davis (2000). Therefore, a similar 7-point scale was used in other questions. Learners’ and facilitators’ responses are compared in Figure 7. The highest rated statements were those regarding the perceived usefulness and perceived ease of use: “I find EduFeedr to be useful in my learning (job)” (M = 5.13; SD = 1.70), “Interacting with EduFeedr does not require a lot of my mental effort” (M = 5.53; SD = 1.32), and “I find EduFeedr to be easy to use” (M = 5.37; SD = 1.54). For almost all the statements, the mean response from the facilitators was higher than from the learners. The only exception was the statement “Interacting with EduFeedr does not require a lot of my mental effort”, which was rated slightly higher by the learners. Australasian Journal of Educational Technology, 2016, 32(2). 75 ascilite Figure 7. Technology acceptance of EduFeedr Web analytics of EduFeedr In order to confirm the findings of the survey, Google Analytics was used to compare the In-Page Analytics from 10 courses in which the respondents participated. The Google Analytics data included all visitors on the courses, as it was not possible to receive separate data about visitors who had answered the survey. In-Page Analytics indicate the percentage and number of clicks for each visited link. Figure 8 shows the In-Page Analytics view for the course feed page in one of the courses. In this particular course, the progress page was the most visited section (15% of clicks) followed by the course feed page (10%) and course information page (4.8%). Fixed menu items received 41.7% of clicks on that page, with the remainder being for the dynamic content – blog posts and comments. In total, the front pages of these courses received 9445 page views during an 8-month period. The average bounce rate for course front pages was 24.2%. The bounce rate refers to the percentage of visitors who leave the site after visiting only the entrance page (Booth & Jansen, 2009). It may be considered a good result since 3 visitors out of 4 continue browsing the course through EduFeedr. On average, visitors spent 1 minute and 9 seconds reading the recent blog posts and comments on the course feed page; the average total length of visit to EduFeedr was 4 minutes and 58 seconds. A total of 13.7% of visitors continued Australasian Journal of Educational Technology, 2016, 32(2). 76 ascilite from the front page to the progress page and 12.6% of visitors went to the participants’ page. It was also noticed that the usage pattern changed according to the learner’s experience with EduFeedr. In courses where the learners had previous experience with EduFeedr, more than 20% of visitors continued to the progress page to compare their progress with other learners and to access the submitted assignments. The progress page was also the second most visited section in the 10 courses (N = 4989). The participants’ page (N = 2712), assignments page (N = 2150) and course information page (N = 1416) received fewer views. The least visited section was the social network visualisation page (N = 696). Figure 8. In-Page Analytics page in Google Analytics highlighting the percentage of clicks for each link Comparing the results of the survey and web analytics Usage statistics from Google Analytics support the main findings of the survey. The survey indicated that EduFeedr enables learners and facilitators to follow recent blog posts and comments to other blogs (see activities A1 and A3 in Figure 6). This is supported by the data from Google Analytics, which shows that the typical visitor spent more than one minute reading the beginnings of recent blog posts and comments on the course feed page. The survey results suggest that the learners have found it useful to compare their progress with other learners (see activity A8 in Figure 6). This is supported by the web analytics data that shows a high number of visits to the progress page. In the survey, the learners rated the awareness of assignments and deadlines more important than communicating with other participants. However, web analytics data shows that the participants’ page was visited more often than the assignments page. The survey showed that the learners found the social network visualisation less useful (see activity A9 in Figure 6). Web analytics data also indicated that the social network page was the least visited section. Discussion This study contributes to research on the use of blogs in education and provides insight into the further design and development of tools for open learning with blogs. The findings show that both learners and facilitators of open online courses consider the activities related to submitting assignments and providing feedback to be the most important. Although this may be influenced by the fact that in the majority of the courses, the students were evaluated and graded based on their blog posts, the results exhibit some characteristics of open online courses using blogs. In these types of courses, both learners and facilitators expect a high level of social interaction among the participants. These results are in line with a number of studies that emphasise the motivating and compelling effect of assessed assignments on blogging (Gray, Thompson, Sheard, Clerehan, & Hamilton, 2010; Henderson, Balatti, Knight, & Haase, 2010) and the usefulness of reading other learners’ ideas and getting feedback on one’s own blog posts (Churchill, Australasian Journal of Educational Technology, 2016, 32(2). 77 ascilite 2009; Ellison & Wu 2008). The study also indicates some areas where online tools for courses using blogs could be improved. These are discussed in more detail below. While the learners and the facilitators were positive about how EduFeedr combines the fragmented discussions, some relatively minor changes to the current features could improve the tool. The course feed page, for example, currently displays a fixed number of the most recent blog posts and comments. This has not been an issue in the current courses because the average number of course participants has been 21. However, it would become an issue in larger courses with very active discussions. This issue could be resolved by loading additional content dynamically when the user is scrolling down. The course feed page could also provide combined RSS feeds for all posts and comments. The ability of EduFeedr to help in the coordination of the course was evaluated relatively high by both the learners and the facilitators. However, a number of improvements could be made to the coordination features as well. EduFeedr relies solely on the data that is aggregated from Atom feeds. In several cases, this data is not sufficient. The two most typical examples of insufficient data are: (a) the link to the assignment specification is missing from the learners’ blog post; (b) the learner who wrote a comment was not logged in to a blogging service. As a solution to the problem, a feature has been designed that allows facilitators to connect blog posts with assignments and comments with an author, using a simple drop-down menu. Having complete data about submitted assignments and written comments would improve the browsing and visualisations of course data. Furthermore, learner profiles that aggregate all the blog posts and comments that the learner has written in the course, would help the coordination of the course. When compared to other features, the facilitators were less positive about how EduFeedr supports grading. In order to support grading the assignments, EduFeedr provides a grading table that the facilitators can download and utilise in their preferred spreadsheet application. Grading could also be improved by having the ability to categorise and annotate the blog posts. For example, Chu, Chan and Tiwari (2012) have proposed a categorisation scheme in which blog posts can be divided into 4 themes (cognitive, metacognitive-reflective, collaborative/social, affective) and 10 sub-themes. This kind of categorisation would provide a further option for browsing the blog posts, as well as helping in the coordination process and creating opportunities for learning analytics. In regard to the awareness of learning activities, learners found the progress chart of EduFeedr to be most useful feature. Designing additional visualisations showing progress and relations of the learners in the course could be useful. For instance, one of the prototypes that the authors have created compares the number of written and received comments for each learner. Some facilitators have a practice of writing summaries of each assignment by reflecting on how the learners did with the assignment. To promote this good practice, EduFeedr could provide a word cloud visualisation (Gottron, 2009) that summarises the content of student posts for each assignment. Features such as these were found to be important in earlier studies as well. For instance, Miyazoe and Anderson (2011) have studied the effect of blog visualisations on student learning. Their study concludes that viewing the online performance of other learners can function as a self-regulatory mechanism and could produce improved learning outcomes. Both the survey results and Google Analytics indicate that the learners and the facilitators found the current social network visualisation less valuable than other sections of EduFeedr. Larusson and Alterman (2009) have studied visualising students blogging activities in a course context where each learner had a blog. One of their visualisations is close to the social network graph in EduFeedr. However, their visualisation also allows highlighting interactions related to one student. This is useful when a large number of connections make the graph cluttered. Redesigning EduFeedr’s social network visualisation by adding different views and the ability to navigate to blog posts could provide additional value for learners and facilitators. The relationship between scripting and awareness is a multifaceted issue that has been studied also in some recent research (Miller & Hadwin, 2015; Rodríguez-Triana et al., 2014; Tsovaltzi, Puhl, Judele, & Weinberger, 2014). Rodríguez-Triana et al. (2014) suggest a process in which monitoring of learner interactions is designed into pedagogical scripts and awareness tools can advice learners based on the collected data. In a simplified way, this process is followed also in EduFeedr (e.g., when learners are asked to include a link to the assignment specification in their blog post and decisions can be made based on progress visualisation). Tsovaltzi et al. (2014) studied how group awareness tools and argumentation scripts influence learning in Facebook. One of their results was that group awareness support of Australasian Journal of Educational Technology, 2016, 32(2). 78 ascilite upcoming argumentative processes could be actually counterproductive for learning. Using argumentation scripts to guide the discussion in blogs would require a more rigid structure. For example, argument types could be specified in comments using hashtags. Miller and Hadwin (2015) discussed how using scripting tools together with group awareness tools could support regulation of collaboration. Combining scripting tools with blogs is an issue that needs further research. From the technical perspective, scripting tools cannot be easily integrated with free blog hosting services that are typically used in open online courses. Also, scripting tools often impose a more defined structure for the collaboration while blog posts and comments have a relatively simple structure. These issues may lead to a danger of over-scripting. In the study, there are two limitations. First, the learners participating in the study had rather different levels of experience with EduFeedr. Some of the learners had participated in courses where all the features of EduFeedr were used extensively. Other learners took courses where the facilitator didn’t specify assignments in EduFeedr and therefore fully lacked the progress chart that is one of the most visited pages in a typical EduFeedr course. Thus, the responses from learners who didn’t use one of the most important features of EduFeedr, may have slightly distorted the overall results. Learners from courses where the assignments feature was used had slightly higher agreement with the statements “EduFeedr helps me to compare my process with other learners” (M = 5.27, for all learners M = 5.08) and “I find EduFeedr to be useful in my learning” (M = 5.18, for all learners M = 5.05). Secondly, the survey method itself is limited in research, which is partly aiming to serve design. In design research one needs also rich qualitative data that is gathered through user observations and interviews. Conclusions This paper addressed four research questions and design challenges that are crucial for designing online tools for open learning with blogs: (a) combining and presenting fragmented discussions, (b) providing coordination structures, (c) supporting awareness, and (d) preventing over-scripting of courses. These questions and challenges stem from the authors’ own research as well as from their review of related research on using blogs in learning (Churchill, 2009; Kim, 2008; Sim & Hew, 2010). In order to study how and to what extent these challenges could be addressed in the design of an online learning tool, the prototype EduFeedr was designed and developed. A survey was conducted to research learner and facilitator perceptions on the importance of these challenges and related learning activities, as well as their satisfaction with how EduFeedr addresses these challenges. A total of 61 learners and 8 facilitators from 10 courses completed the survey. The results of the study confirmed earlier research that identified the fragmentation of discussions and the awareness of learning activities as critical issues in blog-based courses (Kop et al., 2011; Xie et al., 2008). On the other hand, the lack of coordination structures and the danger of over-scripting, also reported as important in earlier studies (Dillenbourg, 2002), were not found to be that critical in this survey. Through the design process of EduFeedr, a number of solutions were found which addressed the issue of fragmentation of discussions. Indeed, learners and facilitators agreed that EduFeedr enabled them to follow recent blog posts and comments. Moreover, the EduFeedr design also addressed the lack of coordination structures, and supported awareness in open learning with blogs; learners and facilitators agreed that EduFeedr enabled them to monitor the submission of assignments and compare progress with other learners. Usage data from Google Analytics supported the survey findings. However, it was concluded that the danger of over-scripting the learning activities is a more comprehensive pedagogical issue that cannot be addressed solely by the design of online learning tools. The main contribution of this study is to highlight the issues that arise in open learning with blogs and to explore ways of addressing these issues in tool design. The authors consider the support of increased awareness on the part of learners and facilitators to be the most promising direction for future research. This is related both to information visualisation and learning analytics of blog-based learning environments. EduFeedr would benefit from additional visualisations that help to compare learner activity in blog-based discussions. More evaluations are needed on the actual use of EduFeedr in different types of blog-based courses. Also, the danger of over-scripting is a complex issue that needs further study. Australasian Journal of Educational Technology, 2016, 32(2). 79 ascilite Acknowledgement This paper is dedicated to the memory of co-author Erik Duval, who passed away on March 12th, 2016. References Berners-Lee, T. (2010). Long live the web: A call for continued open standards and neutrality. Scientific American, 303(6), 80–85. Retrieved from http://www.scientificamerican.com/article/long-live-the- web/ Booth, D., & Jansen, B. J. (2009). A review of methodologies for analyzing websites. In B. J. Jansen, A. Spink, & I. Taksa (Eds.), Handbook of Research on Web Log Analysis (pp. 141–162). Hershey, PA: IGI Global. doi:10.4018/978-1-59904-974-8.ch008 Cakir, H. (2013). Use of blogs in pre-service teacher education to improve student engagement. Computers & Education, 68, 244–252. doi:10.1016/j.compedu.2013.05.013 Carroll, J. M. (2000). Making use: Scenario-based design of human-computer interactions. Cambridge, MA: The MIT Press. Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M. B., & McCrickard, D. S. (2003). Notification and awareness: synchronizing task-oriented collaborative activity. International Journal of Human- Computer Studies, 58(5), 605–632. doi:10.1016/S1071-5819(03)00024-7 Chu, S. K.W., Chan, C. K.K., & Tiwari, A. F.Y. (2012). Using blogs to support learning during internship. Computers & Education, 58(3), 989–1000. doi:10.1016/j.compedu.2011.08.027 Churchill, D. (2009). Educational applications of Web 2.0: Using blogs to support teaching and learning. British Journal of Educational Technology, 40(1), 179–183. doi:10.1111/j.1467-8535.2008.00865.x Cohn, M. (2004). User stories applied: For agile software development. Boston, MA: Addison-Wesley. Dabbagh, N., & Kitsantas, A. (2012). Personal Learning Environments, social media, and self-regulated learning: A natural formula for connecting formal and informal learning. The Internet and Higher Education, 15(1), 3–8. doi:10.1016/j.iheduc.2011.06.002 Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In P. A. Kirschner (Ed.), Three worlds of CSCL: Can we support CSCL? (pp. 61– 91). Herleen: Open Universiteit Nederland. Dillenbourg, P., Zufferey, G., Alavi, H., Jermann, P., Do-Lenh, S., Bonnard, Q., ... Kaplan, F. (2011). Classroom orchestration: The third circle of usability. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting computer-supported collaborative learning to policy and practice: CSCL2011 Conference proceedings (Vol. 1, pp. 510–517). Hong Kong: International Society of the Learning Sciences. Downes, S. (2010). New technology supporting informal learning. Journal of Emerging Technologies in Web Intelligence, 2(1), 27–33. doi:10.4304/jetwi.2.1.27-33 Efimova, L., & de Moor, A. (2005). Beyond personal web publishing: An exploratory study of conversational blogging practices. Proceedings of the 38th Hawaii International Conference on System Sciences (p. 107a). Los Alamitos, CA: IEEE. doi:10.1109/HICSS.2005.118 Ellison, N. B., & Wu, Y. (2008). Blogging in the classroom: A preliminary exploration of student attitudes and impact on comprehension. Journal of Educational Multimedia and Hypermedia, 17(1), 99–122. Retrieved from http://www.editlib.org/p/24310/ Fini, A. (2009). The technological dimension of a massive open online course: The case of the CCK08 course tools. International Review of Research in Open and Distance Learning, 10(5), 1–26. Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/643 Goktas, Y., & Demirel, T. (2012). Blog-enhanced ICT courses: Examining their effects on prospective teachers’ ICT competencies and perceptions. Computers & Education, 58(3), 908–917. doi:10.1016/j.compedu.2011.11.004 Gottron, T. (2009). Document word clouds: Visualising web documents as tag clouds to aid users in relevance decisions. In M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, & G. Tsakonas (Eds.), Research and advanced technology for digital libraries (Vol. 5714, pp. 94–105). Berlin: Springer. doi:10.1007/978-3-642-04346-8_11 Gray, K., Thompson, C., Sheard, J., Clerehan, R., & Hamilton, M. (2010). Students as Web 2.0 authors: Implications for assessment design and conduct. Australasian Journal of Educational Technology, 26(1), 105–122. Retrieved from http://ascilite.org.au/ajet/ajet26/gray.html Australasian Journal of Educational Technology, 2016, 32(2). 80 ascilite Henderson, L., Balatti, J., Knight, C., & Haase, M. (2010). Motivation and participation in learning blogs: challenging the role of assessment. In M. Vick (Ed.). Teacher education for a sustainable future: Proceedings of the 2010 Australian Teacher Education Association national conference, 4-7 July 2010, Townsville, QLD, Australia (pp. 1–7). Townsville: ATEA. Jones, D. (2013, May 12). BIM – Feed aggregation management and marking. Retrieved from http://davidtjones.wordpress.com/research/bam-blog-aggregation-management/ Kim, H. N. (2008). The phenomenon of blogs and theoretical model of blog use in educational contexts. Computers & Education, 51(3), 1342–1352. doi:10.1016/j.compedu.2007.12.005 Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distance Learning, 12(7), 74–93. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1041 Laanpere, M., Pata, K., Normak, P., & Põldoja, H. (2014). Pedagogy-driven design of digital learning ecosystems. Computer Science and Information Systems, 11(1), 419–442. doi:10.2298/CSIS121204015L Larusson, J. A., & Alterman, R. (2009). Visualizing student activity in a wiki-mediated co-blogging exercise. Proceedings of the Extended Abstracts on Human Factors in Computing Systems (pp. 4093– 4098). New York, NY: ACM. doi:10.1145/1520340.1520623 Leinonen, T., Toikkanen, T., & Silfvast, K. (2008). Software as hypothesis: Research-based design methodology. Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (pp. 61–70). Indianapolis, IN: Indiana University. Retrieved from http://dl.acm.org/citation.cfm?id=1795244 Malone, T. W., & Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87–119. doi:10.1145/174666.174668 McLoughlin, C., & Lee, M. J.W. (2007). Social software and participatory learning: Pedagogical choices with technology affordances in the Web 2.0 era. In R. J. Atkinson, C. McBeath, S. K. A. Soong, & C. Cheers (Eds.), ICT: Providing choices for learners and learning. Proceedings of ascilite Singapore 2007 (pp. 664–675). Singapore: Nanyang Technological University. Retrieved from http://www.ascilite.org/conferences/singapore07/procs/mcloughlin.pdf Miller, M., & Hadwin, A. (2015). Scripting and awareness tools for regulating collaborative learning: Changing the landscape of support in CSCL. Computers in Human Behavior, 52, 573–588. doi:10.1016/j.chb.2015.01.050 Miyazoe, T., & Anderson, T. (2011). Viewing and participating: Blog visualization and its learning outcomes in blended learning. Proceedings of the Professional Communication Conference (IPCC) (pp. 1–9). Cincinnati, OH: IEEE. doi:10.1109/IPCC.2011.6087217 O’Donnell, A. M., & Dansereau, D. F. (1992). Scripted cooperation in student dyads: A method for analyzing and enhancing academic learning and performance. In R. Hertz-Lazarowitz, & N. Miller (Eds.), Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 120–141). London: Cambridge University Press. Oravec, J. A. (2003). Blending by blogging: Weblogs in blended learning initiatives. Journal of Educational Media, 28(2–3), 225–233. doi:10.1080/1358165032000165671 Popescu, E. (2014). Providing collaborative learning support with social media in an integrated environment. World Wide Web, 17(2), 199–212. doi:10.1007/s11280-012-0172-6 Rodriguez, O. (2013). The concept of openness behind c and x-MOOCs (massive open online courses). Open Praxis, 5(1), 67–73. doi:10.5944/openpraxis.5.1.42 Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2014). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. doi: 10.1111/bjet.12198 Santos, J. L., Verbert, K., Govaerts, S., & Duval, E. (2013). Addressing learner issues with StepUp!: An evaluation. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 14–22). New York, NY: ACM. doi:10.1145/2460296.2460301 Schmidt, K., & Simone, C. (1996). Coordination mechanisms: Towards a conceptual foundation of CSCW systems design. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5(2–3), 155–200. doi:10.1007/BF00133655 Sim, J. W. S., & Hew, K. F. (2010). The use of weblogs in higher education settings: A review of empirical research. Educational Research Review, 5(2), 151–163. doi:10.1016/j.edurev.2010.01.001 Australasian Journal of Educational Technology, 2016, 32(2). 81 ascilite Snyder, C. (2003). Paper prototyping: The fast and easy way to design and refine user interfaces. San Francisco, CA: Morgan Kaufmann. Tomberg, V., Laanpere, M., Ley, T., & Normak, P. (2013). Sustaining teacher control in a blog-based personal learning environment. The International Review of Research in Open and Distance Learning, 14(3), 109–133. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1397 Tsovaltzi, D., Puhl, T., Judele, R., & Weinberger, A. (2014). Group awareness support and argumentation scripts for individual preparation of arguments in Facebook. Computers & Education, 76, 108–118. doi:10.1016/j.compedu.2014.03.012 Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186–204. doi:10.1287/mnsc.46.2.186.11926 Williams, J. B., & Jacobs, J. (2004). Exploring the use of blogs as learning spaces in the higher education sector. Australasian Journal of Educational Technology, 20(2), 232–247. Retrieved from http://ascilite.org.au/ajet/ajet20/williams.html Xie, Y., Ke, F., & Sharma, P. (2008). The effect of peer feedback for blogging on college students' reflective learning processes. The Internet and Higher Education, 11(1), 18–25. doi:10.1016/j.iheduc.2007.11.001 Corresponding author: Hans Põldoja, hans.poldoja@tlu.ee Australasian Journal of Educational Technology © 2016. Please cite as: Põldoja, H., Duval, E. & Leinonen, T. (2016). Design and evaluation of an online tool for open learning with blogs. Australasian Journal of Educational Technology, 32(2), 64-81.