Australasian Journal of Educational Technology, 2015, 31(1). 67 Learning in the Café: Pilot testing the collaborative application for education in Facebook Josh McCarthy The University of South Australia This paper reports on a pilot study using the Café, the collaborative application for education as an online learning environment within the Facebook framework, for first-year tertiary design students. The Café, a new e-learning application, has been designed based on five principles of user interface design – visibility, usability, relevance, accessibility and interactivity – and developed not only to take advantage of Facebook’s popularity and social qualities, but also to provide institutions with an established, structured and dedicated e- learning environment that meets the needs of contemporary tertiary students and teaching staff. From March to June in 2013, 48 students participated within the e-learning environment, in combination with traditional face-to-face classes including lectures and tutorials. Students were required to submit work-in-progress imagery related to major assignments, and provide feedback and critiques to their peers. The evaluation process of this new e-learning application involved pre- and post-semester surveys providing participating students with the opportunity to critically reflect on the experience during the semester. Introduction Facebook is predominantly known as a hub of social networking activity; however, it is quickly being recognised as a reputable and popular e-learning platform (Bosch, 2009; McCarthy, 2012). Since 2008, Facebook has been successfully implemented as an online learning environment within tertiary education case studies around the world (Irwin, Ball, Desbrow, & Leveritt, 2012; Kenney, Kumar, & Hart, 2013; Kurtz, 2013; McCarthy, 2009, 2010, 2012, 2013a; Rambe, 2012; Ritter & Delen, 2013; Shih, 2011). A key factor behind this revolution is Facebook’s immense popularity. Facebook is a familiar tool, and often a part of students’ daily lives (Duffy, 2011; McCarthy, 2012, 2013a, 2013b). At the time of writing, Facebook has 1.11 billion monthly active users (http://www.statisticbrain.com/facebook-statistics). Furthermore, students can access the site using a range of devices from anywhere in the world. It is also free to use, ensuring students can connect with anyone, including global peers and industry leaders, at any time. This accessibility is often perceived by students as a significant benefit as it can allow increased communication with staff and peers, greater access to course material, connections to industry, and access to collaborative learning partners (Bosch, 2009; Irwin et al., 2012; McCarthy, 2012, 2013a). While Facebook has the potential to promote collaborative learning and student interaction, traditional university online learning environments, such as learning management systems (LMSs), negate such action through their closed-system format (Wang, Woo, Quek, Yang, & Liu, 2012). Students must be enrolled within the specific course in order to access the learning environment, and while this structure is well suited to housing course material, such as lecture notes and tutorials, and managing course-related issues, such as assignment submissions, extension requests and course evaluations, it does not accommodate the beneficial academic and social qualities found in Facebook (Deng & Tavares, 2013; McCarthy, 2013a). Students cannot use their LMS to interact with their global peers, or receive feedback from industry mentors, as these potential partners are not authorised to access it (McCarthy, 2010). LMSs often lack social connectivity and the personal profile spaces which today’s students are familiar with (Mazman & Usluel, 2010). In contrast, students see Facebook as a self-regulated space for individual expression and collaborative learning (Rambe, 2012), and a more conducive environment for communication with staff and peers (Wang et al., 2012). The primary benefits of Facebook as a learning tool arise from its ability to enable participants, both students and staff, to share information, knowledge, and artefacts within a community (McCarthy, 2012). The ability to post content and receive feedback from a wide range of collaborators Australasian Journal of Educational Technology, 2015, 31(1). 68 stands as one of the primary educational benefits of the site (Duffy, 2011; McCarthy, 2013a; Richardson, 2006). Case studies using Facebook as an e-learning environment Facebook was used as a learning environment in the first-year course Imaging Our World in the School of Architecture, Landscape Architecture and Urban Design at the University of Adelaide in 2008. Every 2 weeks, students submitted images to an online gallery in Facebook, and critiqued peers’ submissions (McCarthy, 2009). The gallery topics were broad in nature, and open to the students’ own interpretations, allowing for a wide range of images in each: this produced a concurrently wide range of discussions. The experience showed that the virtual classroom hosted by Facebook provided a platform for students to generate preliminary academic and social interactions with peers in first-year university, while meeting the diverse learning needs and attitudes of Generation Y (McCarthy, 2009). These relationships, however, often stayed embryonic in nature, due to a lack of connection between the real and virtual classrooms. In 2009, a stronger link between the in-class and online learning environments was established. This helped transform emergent online connections into meaningful relationships by supporting virtual communication with face-to-face interaction in the classroom (McCarthy, 2010). In 2010, the virtual environment in Facebook enabled students to produce strong academic results by submitting preliminary design work throughout the semester and receiving constant feedback from a range of sources (McCarthy, 2012). In 2011 and 2012, collaborative international design studios were hosted through Facebook with participating student cohorts from the University of Adelaide, the University of South Australia, Swinburne University, Nanyang Technological University and Penn State University. The online environment in Facebook allowed students to connect with their global peers, learn about new fields of design, and improve their own work via feedback from an ever-growing number of sources (McCarthy, 2012). In a 2010 case study involving 450 first-year students at an English-speaking university in South Africa, Rambe (2012) drew many positive outcomes from the use of Facebook as an online learning environment. Rambe found that Facebook could be used to overcome a first-year student’s sense of psychological powerlessness, by providing students with a voice. Facebook also presented opportunities for student collaboration with both peer-generated content and pedagogical content knowledge, and benefitted students by promoting visibility of common problems that students had within the course curriculum. This in turn allowed course staff to easily recognise any challenges students were experiencing (Rambe, 2012). In a 2011 case study involving 253 students over several year levels at Griffith University, Facebook was used as an online learning environment, allowing students to interact with peers and staff to post content related to associated courses. The study found that the incorporation of Facebook into the learning resources of university courses impacted upon student satisfaction of course delivery, as they were able to integrate academic requirements within a social networking platform that they were both familiar and highly engaged with (Irwin et al., 2012). A pre-semester survey indicated that 93% of participants were existing Facebook users, and that 78% anticipated Facebook would facilitate their learning. A post- semester survey showed that the majority of participants visited the associated page at least once a week; the page enhanced communication and interaction between students and the course instructors; interaction with the Facebook page was easy as students were familiar with the interface; students were able to receive updates and information which may have been missed via other forms of communication; and 76% of participants recommended Facebook be used for future courses (Irwin et al., 2012). In a 2011 case study involving 23 English majors at a technological university in Taiwan, Facebook was used as an online learning environment to facilitate peer interaction and feedback within a small cohort (Shih, 2011). The study outlined that a blended approach to learning, which included online peer assessment within Facebook, generated opportunities for greater learning, and increased student willingness to interact and express their own ideas in writing (Shih, 2011). Australasian Journal of Educational Technology, 2015, 31(1). 69 There are considerable deficiencies, both pedagogical and technical, within learning environments in Facebook, which need to be addressed. Online learning environments within Facebook are commonly created using the group, page, or event applications, or a combination of the three. These applications have not been designed or created specifically to use for e-learning; they have been created to facilitate interaction between social networks, and to act as marketing tools for institutions, businesses and celebrities. Furthermore, Facebook developers have reshaped these applications in terms of their functionality and design, resulting in a complete lack of control over the look and operation of any potential learning environment within the site (McCarthy, 2013b). This lack of continuity and control highlights the need for a dedicated e-learning application within the social networking site (SNS). Facebook’s popularity, social qualities and intuitive interface make it the perfect host site for online learning, while its open accessibility ensures it has the capacity to host national and international collaborative learning partnerships. However, the inconsistent functionality and poor design of its in-built tools negatively affects the overall quality of the learning environment, and as a result can weaken the student experience (McCarthy, 2012, 2013b). An analysis of previous case studies using Facebook as a learning environment has also indicated that there needs to be a separation between students’ social and academic activities. Often, when students submit work, such as comments, images or videos, to an academic forum in Facebook, these posts will appear on their friends’ regular news feed, in turn prompting social commentary from users outside of the student cohort (McCarthy, 2013b). This is a significant problem in using Facebook as an educational tool, as it can impact on a student’s willingness to participate and, as a result, can impact on their performance within the learning environment (McCarthy, 2012; Wang et al., 2012). Figure 1. A screen capture from an online learning environment in Facebook during 2012, created using the page application. Participants’ names have been blurred for anonymity. Figure 1 shows a screen capture from a 2012 online learning environment in Facebook, using the page application (McCarthy, 2013b). The image highlights a poor interface design, which impacts negatively on the look of the learning environment, and its operation. The top posts, in both left and right columns, lack any visual content, showing only text, URLs and link icons. Links to internal events and external Australasian Journal of Educational Technology, 2015, 31(1). 70 websites should come with thumbnail preview images, allowing for a more visually appealing page. The top left-hand post is a link to an internal event, or gallery, within the page. This gallery was created with an associated profile image and contains dozens of images within, submitted by students; however, its preview on the page features no imagery whatsoever. Another activity in the bottom right corner is simply cut off half-way through the associated image. The organisation of the posts is also problematic. The left- hand column is dedicated to posts made by the page administrator, while the right-hand column is dedicated to posts made by any non-administrative participants. While good in theory, this means that if posts are made primarily by the administrator or by non-administrative participants, the page’s content will be lopsided. The page lacks an interactive quality – comments related to the posts in the top right are hidden, while the absence of a live feed associated with the page also leads to a lack of social presence. Participants have to go searching for new posts and information rather than seeing them come up in real time (McCarthy, 2013b). E-learning with other social media tools Since 2007, many other SNSs have been formally and informally utilised within tertiary education, including image-host sites Flickr and DeviantArt, video-hosting sites YouTube and Vimeo, microblogging site Twitter, and visual discovery tools Pinterest and Clipix. Within this group, the successful integration of Flickr, YouTube, Twitter and Pinterest into curriculum has been reported in recent years (Gilbert, Bakhshi, Chang, & Torveen, 2013; Hansen, Nowlan, & Winter, 2012; Robbie & Zeeng, 2008; Sinnappan, 2011). Flickr is an image-hosting website which allows users to share photographs, and host images that they embed in other SNSs. Accessibility to Flickr has improved with the advent of an application that can be used on smart devices. While Flickr is an excellent site for storing and displaying image-based content it lacks the social qualities of Facebook as well as the structure of groups and forums. YouTube is a video-sharing website in which users can upload, view, rate and comment on videos. Registered users can access and analyse data regarding uploads, including number of views, peaks and valleys of view times, as well as generic user demographics. Like Flickr, YouTube can be accessed via phone or tablet through a standalone application, ensuring high availability of content. The ability to embed videos in other sites, as well as the rank and comment features, provides a strong basis for peer-to-peer learning. Pinterest is an SNS which allows users to collate media and categorise content based on specific interests. The site allows users to store images, links and videos and sort them on different pinboards. Aim and research questions While the aforementioned SNSs provide opportunities for e-learning, none provides a complete online learning environment, and there is a clear need for an e-learning application that takes advantage of the popularity and social qualities of Facebook, as well as the content-sharing qualities of Flickr, YouTube, Vimeo and Pinterest. This application should be structured and designed specifically for e-learning; it should incorporate the interactive and community-minded aspects of other successful SNSs; it should negate the closed-system format of LMSs; and it should allow participants to separate their academic and social activities should they wish. In response to these factors, as well as the pedagogical concerns of the design of standard Facebook pages, events and groups, and the continuing lack of control over their operation, a custom Facebook application – the Café, a collaborative application for education – has been designed, developed and pilot tested. This paper reports on the pilot testing of the Café used as an e- learning environment for a student cohort from the University of South Australia. The principal aim of this research was to assess the Café’s effectiveness for providing students with an online environment within which to interact with peers and further their understanding of course material. Within this aim were several research questions: • Does the Café provide students with a learning environment that improves academic performance through peer feedback and critique? • Does the Café provide students with an online learning environment that is easy to use and highly accessible? • Does the Café provide students with an online learning environment that is visually engaging and interactive? Australasian Journal of Educational Technology, 2015, 31(1). 71 • Does the Café provide students with an online learning environment that can be separated from their social activities in Facebook? Designing the Café In order to design the Café, it was essential to first establish any fundamental principles of user interface design, as without a specialised interface, the acceptance rates of computer applications as learning environments can remain low (Fadel & Dyson, 2010). Vaughan (2011) noted, “the best user interface demands the least learning effort,” and as such, the design of the Café balanced technical functionality and aesthetic quality to create a system that was not only operational but also engaging, usable and flexible to evolving user needs (McCarthy, 2013b). In 2003, Lockwood and Constantine established a usage-centred design model where the user interface design for online learning environments is focused on user intentions and usage patterns. Within this model, user interface design is determined by the user’s expected experience. The resulting design concentrates on ease of use, visual impact, and the clear organisation of information. This model breaks user interface design into a series of six key principles: structure, simplicity, visibility, feedback, tolerance and reuse. In 2010, Fadel and Dyson established a similar model, identifying four design principles which enhance user interface design: simplicity, emphasis, organisation, and modularity. Within this theory, user interface designs are successful when they are simple, consistent and visually engaging. Fadel and Dyson (2010) also noted that the learning environment should be highly accessible, and prior experience with a specific media or tool can help the user feel more comfortable within the environment. When users feel comfortable within the environment they often interact more with other participants, supporting the notion that Facebook is a suitable host site for such a learning environment. The user interface should also promote interaction between participants (Lee, Danis, Miller, & Jung, 2001), and encourage participants to become active and increase their contribution (Erickson and Kellogg, 2000; Girgensohn & Lee, 2002; Lee & Boling, 1999; Szabo & Kanuka, 1999). Lastly, the interface should be easy to use and navigate. A well-designed learning environment should provide users with a convenient interface with features including ease of use, ease of understanding, and helpfulness (Wei, Chen, & Kinshuk, 2012). The theories presented by Lockwood and Constantine (2003), and Fadel and Dyson (2010), suggest that a simple user interface design, featuring visually engaging imagery, and accompanied by high accessibility and a popular platform, will lead to a connection between the users and the online learning environment. This connection can be referred to as social presence. Social presence is the degree of feeling, perception, and reaction to another intellectual entity in an e-learning environment (Tu & McIsaac, 2002). One significant barrier to supporting interaction in current e-learning environments is the low sense of social presence experienced by participants compared with a face-to- face environment (Fadel & Dyson, 2010). This can be particularly evident with LMSs where there is often a lack of meaningful interaction between participants, both students and staff (DeSchryver, Mishra, Koehler, & Francis, 2009; Dunlap & Lowenthal, 2009). However, SNSs, such as Facebook, have the capacity to construct social presence by providing students, particularly introverted students, with a learning environment within which they can easily and comfortably engage with their peers (Kawachi, 2013). Social presence is a predictor of perceived learning outcomes and learner satisfaction (Gunawaredena & Zittle, 1997; Whipp & Lorentz, 2009), and can be constructed via co-presence, intimacy and immediacy (Wei et al., 2012). Co-presence refers to students learning together within a cohort; intimacy refers to any involvement or interaction with other students within the cohort; and immediacy refers to the timeliness and intensity of such interaction with other students within the cohort. Social presence has significant effects on learning interaction, which in turn has significant effects on learning performance (Bolliger, 2004; Ferguson, 2010; Jin, 2010; Picciano, 2002; Richardson & Swan, 2003; Tu & McIsaac, 2002; Wei et al., 2012). Based on the principles of user interface design presented by Lockwood and Constantine (2003), and Fadel and Dyson (2010), as well as the key values of social presence, a new set of five principles of user interface design has been established for the creation of the Café: visibility, usability, relevance, accessibility and interactivity, as outlined in Table 1. Australasian Journal of Educational Technology, 2015, 31(1). 72 Table 1 The five principles of user interface design for the design of the Café Principle Design requirements Café features Visibility The design should provoke immediate interest within the user without distracting them with unnecessary information. The design should make users aware of other participants’ activities. Customisable banner image ensures immediate visual interest; Image-based navigation pane and grid layout; Embedded videos from video-hosting sites; Full image thumbnail previews, and full resolution images and full screen videos when viewed individually; Pin-to-top functionality for forum managers ensures important content is instantly visible Usability The design should provide a strong structure to the user interface, adopting clear design principles, using navigational elements recognisable to users, and putting related things together and separating unrelated things. The design should promote interaction between users. Image and text-based navigation pane ensure ease of use; Help documents for each section of the application provide support when required; Feedback forms allow participants to report bugs and suggest changes Relevance The design should provide relevant functionality to the intended end users, allowing for content-based submissions, and communication between participants. Customisable banner image to suit the content of the forum; Content submissions on the pinboard and in galleries; Communication with other participants via commenting tools; Accessibility The design should promote accessibility via a range of mediums including desktop and personal computers, and smart devices. Accessibility should in turn enable high levels of interaction between participants. The Café can be accessed via any device with an Internet connection; The live feed allows participants to immediately view new content within a forum; The Café can be accessed via Facebook and its official website http://www.thecafeapp.com/app Interactivity The design should promote interaction and a sense of social presence within the learning environment, allowing for both synchronous and asynchronous communication between participants. Participants can interact with their peers by posting content, commenting on content, liking content, asking questions and providing responses; Participants can add, collate and categorise content in their myCafe page. These five principles in turn lead to the formation of a hypothesis model for the design of the Café learning environment, as shown in Figure 2 (McCarthy, 2013b). Facebook’s immense popularity, social qualities and intuitive interface are utilised, by adopting the SNS as the host site for the online learning environment. Focus is placed upon the five established principles of user interface design, which lead to an acceptance of the learning environment, and the formation of social presence within the student cohort. This in turn promotes increased interaction between participants – including peer-to-peer feedback and critiques – and a stronger engagement with course material. Students benefit from the increased feedback on work-in-progress – receiving critiques from a wide range of sources, rather than just the course coordinator – and produce stronger results in course assessment as a result. Finally, the combined increase in (a) participant interaction, (b) engagement with course material, and (c) improved academic performance, all lead to an enhanced student experience within the course (McCarthy, 2013b). Australasian Journal of Educational Technology, 2015, 31(1). 73 Figure 2. The hypothesis model for enhancing the student experience through a custom designed online learning environment hosted by Facebook (McCarthy, 2013b) The Café can be accessed through both Facebook and its dedicated website, http://www.thecafeapp.com/app, and can be used on a range of devices. The Café allows a user to establish an online learning environment in the form of a forum and invite participants to join. A forum can be both open and closed format, at the discretion of the forum manager, enabling the creation of both private and public forums. The forum manager can also change this setting at any stage, allowing open access or closed access to forums at different times. Within the forum there are four key areas: the pinboard, galleries, Q and A, and MyCafe. The pinboard acts as the home page for the forum. On the pinboard, participants are able to pin images, videos, comments and links, relevant to the forum. All image, video and link-based posts contain imagery to create a more visually engaging online space, while a live feed tracks all submissions within the entire forum in real time. The galleries allow the forum manager to establish virtual gallery spaces – content pages for student submissions. Forum managers can outline the details of a gallery, such as opening and closing times, and content descriptions. Participants can then submit content, be it image, video, text or link-based. Once content is submitted to a gallery, participants can comment on, like or add the submission to their personal space within the forum in myCafe. The Q and A page acts as a discussion board for the forum, and provides participants with the opportunity to ask questions and provide responses. The final section within the application is myCafe. This acts as the participants’ personal space within the Café. Participants are able to collate all of their submissions within the forum. They are also able to collate, and organise into categories, submissions from other participants within the forum, as well as submissions they comment on. This supports the student by (a) facilitating personal reflection on their work and (b) helping them to prepare for exams. It also allows the student to develop an online collection of precedents and examples to enhance their broader knowledge of the topics delivered within a course. Lastly it makes assessing the student’s performance, interaction and engagement within the online learning environment much easier for associated staff by collating all of their submissions in one place. Participants maintain ownership of all of their submitted original content. All data is stored in the application database on the Café’s web server. Method In 2013, 48 students from the first-year design course, Design Language in Media Arts (DLMA), in the Bachelor of Media Arts program at the University of South Australia participated in the pilot study. DLMA introduces students to design theory in a broad range of fields including film, performance, animation and digital design. In the first lecture of the semester, students were introduced to the e- learning environment in the Café, and shown how to access, install and use the application. Following this introduction, students were given the opportunity to take part in an anonymous pre-semester survey, hosted online via Survey Monkey, to determine their expectations of the learning experience ahead. The survey included three broad types of measures: demographic data, students’ attitudes towards online learning environments in Facebook, and students’ attitudes towards in class and online participation. In total, 39 students participated in the survey, a response rate of 81%. The breakdown of student demographics within the cohort is outlined in Table 2. Australasian Journal of Educational Technology, 2015, 31(1). 74 Table 2 The breakdown of student demographics within the cohort Demographic Number of students in DLMA Percentage of cohort Number of respondents Percentage of respondents within each demographic Number of respondents 48 100% 39 81% Gender Male 23 48% 18 78% Female 25 52% 21 84% Student type Local student 38 79% 31 82% International student 10 21% 8 80% Age 17–18 8 17% 7 88% 19–24 31 65% 24 77% 25–34 4 8% 3 75% 35+ 5 10% 5 100% The survey contained four questions related to e-learning in Facebook. 33% of students indicated they had used Facebook for some form of online learning in the past, and 69% indicated they believed it was an effective host site for such activities. 64% of students indicated it was important to keep academic and personal activities on Facebook separate, and 67% said they were looking forward to using the Café learning environment during the semester. Mean response and broad agreement data are collated in Table 3 and Table 4. Table 3 Student responses to questions and statements Topic Yes No Have you used Facebook for any form of online or collaborative learning in the past? 33% 67% Topic In class Online I prefer asking questions 42% 58% I prefer critiquing my classmates’ work 14% 86% Engaging in academic discussions 46% 54% Table 4 Student responses to questions and statements. Topic Mean response Broad agreement Undecided Broad disagreement I believe Facebook is an effective host site for an online and collaborative learning environment. 3.90 69% 26% 5% I believe academic and personal activities in Facebook should be kept separate. 3.77 64% 31% 5% I am looking forward to using the Café learning environment within Facebook this semester. 3.85 67% 31% 2% Note. The survey used a 5-point Likert scale from 1 (strongly disagree), to 3 (undecided), to 5 (strongly agree). Students also provided open-ended responses to the topics raised within the questionnaire. Participants who believed Facebook was an effective host site for online and collaborative learning noted its familiarity and popularity as key reasons: Facebook is a website that many people are familiar with, which could allow students to adapt to the learning environment quicker. Australasian Journal of Educational Technology, 2015, 31(1). 75 Facebook is a site which most people use and has become an easy way for students to collaborate without being physically together. Students looking forward to using the Café in Facebook during the semester, cited its design, layout, and originality as factors: The Cafe has been developed for use in an environment that many people are familiar with and are already able to access. The UI of the application itself is intuitive and easy to use, and the ability to access the app on a mobile device makes it easy for students to upload work when needed. It's something new and different. It's very exciting. However, not all students were enamoured with the concept, with 31% undecided if they were looking forward to using the Café learning environment in Facebook: I'm not convinced yet as I see Facebook as a total distraction and I avoid it. I'm worried I'll forget to submit something because I'll get distracted while logging on by something cool on the newsfeed. Many students wanted academic and personal activities in Facebook to be kept separate, both to avoid academic-related content coming up in their friends’ news feeds, and to avoid their personal activities being linked to their studies: If a student does not wish to share regular Facebook activity with students in an academic learning environment, they should have the option to filter that information, if they wish. I'd prefer to keep all my submissions separate as I don't feel like I need to share my work with everyone on Facebook. Utilising the Café During the semester students were required to regularly submit work-in-progress imagery related to major assignments, and provide feedback and critiques to their peers. Participation within the Café was worth 15% of the final grade for the course, and students were assessed on three key components: (a) the quality of the submitted imagery, (b) the descriptions that accompanied the submitted imagery, and (c) the quality and consistency of their peer critiques. Students were provided with clear directions as to what constituted an appropriate critique to ensure constructive feedback and to eliminate elementary comments such as ‘cool’ or ‘LOL’. Correct punctuation, spelling, and grammar were assessed, and the depth of the critique was considered. Students were required to provide comprehensive analysis of the submission and consider how the work could be revised or improved. Over 13 weeks, 48 students visited the Café forum 2544 times at an average of 53 visits per student, or just over four visits per student per week. There were 79 posts to the pinboard, including comments, links, images and videos. 371 posts were made to six content galleries, which received a total of 641 comments. 22 questions were raised within the Q & A page, which received a total of 37 responses. Lastly, 173 posts were added by students to their respective myCafe pages. Figure 3 (left) shows a screen capture from the forum’s pinboard, accessed on 1/10/2013. The screen capture shows a custom banner at the top of the page, created by a student as part of a design competition within the course, under which is the forum navigation. Below that is the live feed showing forum activity in real time. Posts from participants are stored at the bottom of the page in three columns. Posts move left to right and top to bottom, as new content is submitted. This ensures new content is always at the top of the pinboard. Figure 3 (right) depicts a screen capture from a forum gallery. The screen capture shows the gallery profile image and description at the top of the page, followed by thumbnail previews of student Australasian Journal of Educational Technology, 2015, 31(1). 76 submissions. Individual submissions can be viewed in full screen, and can be liked, commented on, and added to myCafe. Figure 3. Left: a screen capture of the forum pinboard; right: a screen capture of a forum gallery Figure 4 (left) shows a screen capture from the forum’s Q & A page, accessed on 30/04/2013. The screen capture shows a list of questions asked by participants in reverse chronological order, ensuring new questions are at the top of the page. Any participant can provide a response to a question. Figure 3 (right) shows a screen capture from a student’s myCafe page. Content within the myCafe page is arranged in three columns: my posts – every post made by the participant; commented posts – every post which the participant has commented on; and added posts – every post which the participant has chosen to add to myCafe. Figure 4. Left: a screen capture of the Q & A page; right: a screen capture of a student’s myCafe page Australasian Journal of Educational Technology, 2015, 31(1). 77 Student responses to the learning environment within the Café The student experience during the semester was evaluated through an online, 10-question, post-semester survey. Students were invited to take part in the survey over a 3-week period at the end of the semester. Both of the pre- and post-semester questionnaires were designed and distributed under the approval from the Human Research Ethics Committee at the University of South Australia, prior to the commencement of the online learning environment in the Café. The post-semester questionnaire addressed the design and functionality of the application, the perceived effectiveness of the learning environment, and the students’ experiences throughout the semester. 39 students participated in the survey, again a response rate of 81%. Participants were given the opportunity to assess the learning experience in the form of Likert-scale statements and open-ended questions. Mean response and broad agreement statistics, related to the 6 Likert-scale statements are shown in Table 5. Table 5 Student responses to questions and statements from the post-semester survey Topic Mean response Broad agreement Undecided Broad disagreement I would like to use the Café as an online learning environment again in future courses. 4.54 98% 2% 0% During the semester I received beneficial feedback through the Café galleries. 4.33 87% 13% 0% During the semester the Café promoted interaction with peers. 4.28 92% 6% 2% Having all of my posts collated in myCafe was beneficial. 4.13 90% 10% 0% The ability to collate and categorise other students’ posts in myCafe was beneficial. 4.03 79% 21% 0% During the semester the Café generated rewarding academic discussions. 4.00 77% 23% 0% The questionnaire outlined student reactions towards the online learning environment in the Cafe. 87% of participants indicated they received beneficial feedback from their peers during the semester while 77% noted the learning environment generated rewarding academic discussions. Responses included: The Cafe is a great environment for promoting student discussion and sharing of ideas and thoughts on other students' work. It was good to hear different perspectives on the work I uploaded and to be able to provide feedback for other students, as well. I found the Café to be a useful way to easily create conversation and discuss ideas and feedback about my work. I liked how easy it was to contact the tutor though the Café and how they posted helpful information and videos on the pinboard. I didn’t receive many comments at the start of semester, which was disappointing. It was good to get feedback for the last assignment though. 92% of participants indicated the Café promoted interaction with their peers, by providing students with a familiar and accessible online environment: Without the Café app, there would probably be less natural inclination among the students to discuss each other’s work, but because the Café operates through Facebook, which is an environment that a lot of people are familiar with, I think people felt more comfortable with discussing their thoughts and ideas with each other. Offering students design activities beyond the course assessment was also instrumental in promoting interaction between peers online: Australasian Journal of Educational Technology, 2015, 31(1). 78 The banner contest and 50 word comment rule are very peer interactive. There was always something to do or discuss during the semester on the Café. 90% of participants responded positively towards myCafe, the personalised space within the Café, citing the ability to collate both posts from the pinboard, their own work and also peers’ posts from the galleries: It was helpful to collect the posts from the pinboard - images and videos related to assignments or lectures. Being able to locate all my work in one spot was useful and saved a lot of time browsing through individual galleries. 98% of participants indicated they wanted to use the Café as a learning environment in the future, referring to the application’s ability to generate peer discussions, to learn about design, and to promote group learning: I think it's worked well as a platform for uploading work for peer review and for generating some well thought out and provoking discussion. Definitely - this was a good way to get everyone learning together. Yes this was a great system. Much better than learnOnline. Attitudes within the cohort towards in-class and online interaction between peers changed during the semester. 74% of participants indicated they preferred engaging in academic discussions online, up 20% from the pre-semester survey. 95% stated they preferred critiquing their classmates’ work online, up 9%; however, only 47% said they preferred asking questions online, dropping 11%, as outlined in Table 6. Table 6 Student responses regarding online and in class interaction, from the post-semester survey Topic Pre-semester Post-semester Difference In class Online In class Online In class Online I prefer asking questions. 42% 58% 53% 47% +11% -11% I prefer critiquing my classmates’ work. 14% 86% 5% 95% -9% +9% Engaging in academic discussions 46% 54% 26% 74% -20% +20% Some students noted that an online environment gave them more time to think and react: The benefit of doing these things online as opposed to during a lecture or in class is that there's more freedom for people to ask complete questions and for people to be able to give complete answers, in their own time. As an international student it is hard sometimes to discuss in class. It is great to get so many comments from other students and staff. I prefer this type of discussion online as it is documented and I can refer to it accurately later. The majority of participants responded positively when asked about the design, layout and navigation of the Café: Great, simple to navigate and understand. A good layout. A great tool to see peers’ work. Australasian Journal of Educational Technology, 2015, 31(1). 79 Easy navigation - the design is simple to follow and looks more professional than a Facebook page. There were, however, some operational problems with the learning environment throughout the semester, most notably PHP-related loading errors, and some long loading times: Sometimes the pages took a while to load because they contained large files. Sometimes uploading work or comments has been problematic throughout the semester, due to PHP errors. The only thing that I find annoying is that I have to enter the DLMA forum when I launch the Café instead of it taking me straight there. Participants were asked to make suggestions for revising the learning environment, and most responses focussed on faster load times and a specific smartphone app: Would be good to have a custom app on iPhone. Make it faster to access. It would be better if there is a notification on Facebook whenever someone posts, 'likes', and comments in the Café. Final reflections on the learning experience provided insight into how the Café could grow in the future: As it is, the Café is a functional application and is impressive for its first iteration. If the code problems are ironed out and the app is updated regularly as a result of student feedback through surveys such as these, the Café can evolve and become a very useful tool that could be implemented across other courses. The Café Facebook environment is an excellent initiative that is worth considering for wider implementation across courses. At the start of the semester I was very sceptical in regards to the whole 'Facebook idea', but it actually worked really well. It is almost surprising for someone like me who still uses a very old Nokia without internet! Student performance within the course At the completion of the semester student performance was analysed to help establish whether the online learning environment in the Café had any impact on academic achievements. Assessment within the course was broken down into two categories: 85% of assessment was associated with project work, and 15% of assessment was associated with participation in the Café forum. Figure 6 shows the distribution of student grades within the course, charting overall grades, Café participation grades, and project work grades. Grades are categorised as high distinction (HD), distinction (D), credit (C), pass (P) and fail (F). Of most interest is the students’ performance within the project work (major assignments). The Café online learning environment has been created and used to provide students with feedback on work-in- progress so they can produce stronger results within their major projects. For the project work assessment, one student received a high distinction, 20 students received a distinction, 14 students received a credit, 10 students received a pass, and 3 students received a fail grade. Australasian Journal of Educational Technology, 2015, 31(1). 80 Figure 6. Student academic performance within Design Language in Media Arts Figure 7 outlines the average number of visits to the Café during the semester for students within each of the five grade brackets, and shows that higher achieving students visited the application more often. This number increased consistently per grade bracket, from 15.71 visits within the fail bracket, through to 120.00 visits within the high distinction bracket. Figure 7. Average number of visits to the Café per student grade bracket Figure 8 outlines the average number of posts made to the Café galleries during the semester for each of the five grade brackets, and shows again that the higher achieving students posted more content to the Café galleries. The average number of posts ranged from 1.43 within the fail bracket to 14.00 within the high distinction bracket. 0 5 10 15 20 25 HD D C P F N um be r of s tu de nt s Grade Overall grades (100%) Project grades (85%) Café grades (15%) 120.00 65.90 44.14 37.80 15.71 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 HD (1) D (20) C (14) P (10) F (3) N um be r of v is it s pe r st ud en t Grade bracket (number of students) Australasian Journal of Educational Technology, 2015, 31(1). 81 Figure 8. Average number of posts to the Café galleries per student grade bracket Figure 9 outlines the average number of comments made on peers’ submissions in the Café galleries during the semester. Unlike the first two categories, this number did not increase uniformly across the grade brackets. On average, students within the credit bracket posted more comments on peers’ submissions, at 16.07 per student. Figure 9. Average number of comments made on peers’ submissions in the Café galleries per student grade bracket Figure 10 outlines the average number of comments from peers received during the semester. The number of comments received within the Café galleries increased per grade bracket, from 0.71 within the fail bracket, through to 45.00 within the high distinction bracket. 14.00 8.70 7.71 6.50 1.43 0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00 HD (1) D (20) C (14) P (10) F (3) N um be r of p os ts p er st ud en t Grade bracket (number of students) 16.00 14.45 16.07 9.90 1.71 0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00 18.00 HD (1) D (20) C (14) P (10) F (3) N um be r of c om m en ts m ad e pe r st ud en t Grade bracket (number of students) Australasian Journal of Educational Technology, 2015, 31(1). 82 Figure 10. Average number of comments received from peers during the semester per student grade bracket Lastly, Figure 11 outlines the average number of comments from peers received per post. This number increased per grade bracket, from 0.50 within the fail bracket, through to 3.21 in the high distinction bracket. Figure 11. Average number of comments received from peers per post per student grade bracket Discussion The majority of students responded positively towards the learning experience within the Café at the conclusion of the pilot study. Students noted the professional design and simple navigation as two key features of the application, while its accessibility (via Facebook) and interactivity (through posting, commenting on, and collating content) were also popular, outlining the importance of strong user interface design. The five principles of user interface design incorporated into the Café – visibility, usability, relevance, accessibility and interactivity – were repeatedly referred to when students commented upon the positive factors of the learning environment. This was a strong result considering 31% of students were unsure if they were looking forward to using the Café learning environment at the start of semester. At the end of the semester there was still a group of students who were undecided in their appraisal of certain aspects of the Café. While 98% of participants indicated they wanted to use the application again, 23% of students indicated they were unsure whether the Café generated rewarding academic discussions while 21% of students indicated they were undecided whether the ability to collate and categorise other students’ posts in myCafe was beneficial. While these students weren’t specifically negative towards 45.00 17.55 12.79 6.10 0.71 0.00 10.00 20.00 30.00 40.00 50.00 HD (1) D (20) C (14) P (10) F (3) N um be r of c om m en ts re ce iv ed p er s tu de nt Grade Bracket (number of students) 3.21 2.01 1.66 0.94 0.50 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 HD (1) D (20) C (14) P (10) F (3) N um be r of c om m en ts re ce iv ed p er p os t Grade bracket (number of students) Australasian Journal of Educational Technology, 2015, 31(1). 83 these elements within the learning experience, it does raise questions as to why they didn’t respond positively. One student indicated that there was a lack of feedback from peers at the start of semester and this could have impacted on their attitude. In the future, it would be beneficial to ensure there is a mechanism in place to ensure all students receive feedback throughout the entire semester. This may require the course coordinator to monitor feedback levels on all submissions, and provide feedback on submissions that are lacking comments from students. Focus groups could also be brought into future evaluations of the e-learning environment, to attain more detailed feedback, as no other negative comments were received via the survey. The Café enabled students to keep their personal and academic activities throughout the semester separate in Facebook. Whenever a student posted content in the Café, it didn’t appear on their Facebook home page or their friends’ news feeds; it was only visible within the Café. Analysis of student performance within the course suggests there may be a correlation between high levels of participation in the Café and strong academic performance. In four of the five categories analysed – number of visits, number of posts, number of comments received during the semester, and number of comments received per post – the average figure increased per student grade bracket. While the highest achieving students in the course didn’t provide the most comments, they did, on average, visit the learning environment more often, post more content, and receive more comments from their peers. Although this pilot study produced promising results in this respect, the research is currently limited to a single cohort, and it is therefore not possible to establish conclusively that the online learning environment in the Café positively impacted on students’ performance. The research must expand in the future to include additional student cohorts nationally and internationally. There were times during the semester when sections of the learning environment took a long time to load, most notably the gallery pages. This was due to the high number of submissions that was made to each gallery. To combat long load times, pagination was introduced, separating the galleries into pages of ten posts. Gallery load times were further improved through implementing a ‘load when required’ system, where posts only loaded when the user scrolled down the page, and by creating an automated system between the Café and Facebook, where new content was automatically updated twice a day. Another problem that impacted on the learning environment was a PHP error that occasionally occurred when launching the application, most regularly during the first 3 weeks of the semester. It was determined that this error was caused by Facebook’s requirement for third-party cookies to be enabled. These cookies are often blocked in Internet browsers, such as Internet Explorer, causing a page not to load properly. To tackle this issue, instructions to enable cookies were included on the welcome page. Reporting of this error from students diminished significantly after these instructions were added. Research and development of the Café will continue in the future and forthcoming publications will focus on revisions to the design and operation of the application, as well as additional case studies using it as an e-learning environment. It is expected that from mid-2014 the Café will be available to download for free and be used by anyone with an active Facebook account, on any device that has an Internet connection. References Bolliger, D. (2004). Key factors for determining student satisfaction in online courses. International Journal on E-Learning, 3(1), 61–67. Bosch, T. E. (2009). Using online social networking for teaching and learning: Facebook use at the University of Cape Town. Communicatio: South African Journal for Communication Theory and Research, 35(2), 185-200. doi:1080/02500160903250648 Deng, L., & Tavares, N. (2013). From Moodle to Facebook: Exploring students’ motivation and experiences in online communities. Computers & Education, 68, 167-176. doi:10.1016/j.compedu.2013.04.028 DeSchryver, M., Mishra, P., Koehler, M., & Francis, A. (2009). Moodle vs Facebook: Does using Facebook for discussions in an online course enhance perceived social presence and student interaction? In Proceedings of the Society for Information Technology & Teacher Education International Conference 2009 (pp. 329-336). Chesapeake, VA: AACE. Australasian Journal of Educational Technology, 2015, 31(1). 84 Duffy, P. (2011). Facebook or Faceblock: Cautionary tales exploring the rise of social networking within tertiary education. In C. McLoughlin & M. Lee (Eds.), Web 2.0 based e-learning: Applying social informatics for tertiary teaching (pp. 284-300). Hershey, PA: IGI Global. doi:10.4018/978-1-60566- 294-7.ch015 Dunlap, J. C., & Lowenthal, P. R. (2009). Tweeting the night away: Using Twitter to enhance social presence. Journal of Information Systems Education, 20(2), 129-136. Erickson, T., & Kellogg, W. A. (2000). Social translucence: An approach to designing systems that support social processes. ACM Transactions on Computer-Human Interaction, 7(1), 59–83. doi:10.1145/344949.345004 Fadel, L. M., & Dyson, M. C. (2010). Interface design for social interaction in e-learning environments. In A. Tzanavari & N. Tsapatsoulis (Eds.), Affective, interactive and cognitive methods for e-learning design: Creating an optimal education experience (pp. 72-93). Hershey, PA: IGI Global. doi:10.4018/978-1-60566-940-3.ch004 Ferguson, R. (2010). Peer interaction: the experience of distance students at university level. Journal of Computer Assisted Learning, 26(6), 574–584. doi:10.1111/j.1365-2729.2010.00386.x Gilbert, E., Bakhshi, S., Chang, S., & Terveen, L. (2013). “I need to try this!”: A statistical overview of Pinterest,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, ACM, 2427–36. Retrieved from http://www.cc.gatech.edu/~sbakhshi/gilbert.chi13.pinterest.pdf Girgensohn, A., & Lee, A. (2002). Making web sites be places for social interaction. In CSCW’02, Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (pp. 136- 145). New York, NY: ACM. doi:10.1145/587078.587098 Gunawardena, C. N., & Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. American Journal of Distance Education, 11(3), 8-26. doi:10.1080/08923649709526970 Hansen, K., Nowlan, G., & Winter, C. (2012). Pinterest as a Tool: Applications in Academic Libraries and Higher Education. Partnership: The Canadian Journal of Library and Information Practice and Research, 7(2). Irwin, C., Ball, L., Desbrow, B. & Leveritt, M. (2012). Students’ perceptions of using Facebook as an interactive learning resource at university. Australasian Journal of Educational Technology, 28(7), 1221-1232. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index Jin, S. A. A. (2010). Parasocial interaction with an avatar in second life: A typology of the self and an empirical test of the mediating role of social presence. Presence-Teleoperators and Virtual Environments, 19(4), 331–340. doi:10.1162/PRES_a_00001 Kawachi, P. (2013). Online social presence and its correlation with learning. International. Journal of Social Media and Interactive Learning Environments, 1(1), 19–31. doi:10.1504/IJSMILE.2013.051653 Kenney, J., Kumar, S., & Hart, M. (2013). More than a social network: Facebook as a catalyst for an online educational community of practice. International Journal of Social Media and Interactive Learning Environments, 1(4), 355-369. doi:10.1504/IJSMILE.2013.057467 Kurtz, G. (2013). Facebook group as a space for interactive and collaborative learning. International Journal of Social Media and Interactive Learning Environments, 1(4), 406-418. doi:10.1504/IJSMILE.2013.057461 Lee, A., Danis, C., Miller, T., & Jung, Y. (2001). Fostering social interaction in online spaces. In M. Hirose (Ed.), Human–Computer Interaction, INTERACT 2001 (pp. 59-66). Amsterdam: IOS Press. Lee, S. H., & Boling, E. (1999). Screen design guidelines for motivation in interactive multimedia instruction: A survey and framework for designers. Educational Technology, 39(1), 19-26. Lockwood, L., & Constantine, L. (2003). Usability by inspection: Collaborative techniques for software and web applications. In L. Constantine (Ed.), Performance by Design: Proceedings of forUSE 2003, Second International Conference on Usage-Centered Design. Rowley, MA: Ampersand Press. Mazman, S. G., & Usluel, Y. K. (2010). Modeling educational usage of Facebook. Computers and Education, 55(2), 444-453. doi:10.1016/j.compedu.2010.02.008 McCarthy, J. (2009). Utilising Facebook: Immersing Generation-Y students into first year university. Journal of the Education Research Group of Adelaide, 1(2), 39-49. doi:10.1504/IJSMILE.2013.057460 McCarthy, J. (2010). Blended learning environments: Using social networking sites to enhance the first year experience. Australasian Journal of Educational Technology, 26(6), 729-740. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index Australasian Journal of Educational Technology, 2015, 31(1). 85 McCarthy, J. (2012) International design collaboration and mentoring for tertiary students through Facebook. Australasian Journal of Educational Technology, 28(5), 755-775. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index McCarthy, J. (2013a). Learning in Facebook: First year tertiary student reflections from 2008 to 2011. Australasian Journal of Educational Technology, 29(3), 337-356. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index McCarthy, J. (2013b). The Café: creating the ‘collaborative application for education’; a dedicated e- learning environment in Facebook. International Journal of Social Media and Interactive Learning Environments, 1(4), 419-438. doi:10.1504/IJSMILE.2013.057460 Picciano, A. G. (2002). Beyond student perceptions: Issues of interaction, presence, and performance in an online course. JALN, 6(1), 21-40. Rambe, P. (2012). Critical discourse analysis of collaborative engagement in Facebook postings. Australasian Journal of Educational Technology, 28(2), 295-314. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index Richardson, J. C., & Swan, K. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. JALN, 7(1), 68–88. Richardson, W. (2006). Blogs, wikis, podcasts, and other powerful tools for classrooms. Thousand Oaks, CA: Sage. Ritter, N., & Delen, E. (2013). Undergraduates’ Facebook use: Evidence-based practice to implement social media in education. International Journal of Social Media and Interactive Learning Environments, 1(4), 387-405. doi:10.1504/IJSMILE.2013.057463 Robbie, D., & Zeeng, L. (2008). Engaging student social networks to motivate learning: Capturing, analysing and critiquing visual image. International Journal of Learning, 15(3), 153-160. Shih, R.-C. (2011). Can Web 2.0 technology assist college students in learning English writing? Integrating Facebook and peer assessment with blended learning. Assessing students’ Web 2.0 activities in higher education. Australasian Journal of Educational Technology, 27(5), 829-845. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/index Sinnappan, S. (2011, July). Usage of Twitter in class to enrich student interaction and experience. Paper presented at the Social Media in Higher Education Conference, Melbourne, Australia. Szabo, M., & Kanuka, H. (1999). Effects of violating screen design principles of balance, unity, and focus on recall learning, study, time, and completion rates. Journal of Educational Multimedia and Hypermedia, 8(1), 23–42. Tu, C., & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. The American Journal of Distance Education, 16(3), 131-150. doi:10.1207/S15389286AJDE1603_2 Vaughan, T. (2011). Multimedia: Making it work (8th ed.). New York, NY: McGraw-Hill. Wang, Q. Y., Woo, H. L., Quek, C. L., Yang, Y. Q., & Liu, M. (2012). Using the Facebook group as a learning management system: An exploratory study. British Journal of Educational Technology, 43(3), 428-438. doi:10.1111/j.1467-8535.2011.01195.x Wei, C., Chen, N., & Kinshuk. (2012). A model for social presence in online classrooms. Educational Technology Research and Development, 60(3), 529-545. doi:10.1007/s11423-012-9234-9 Whipp, J. L., & Lorentz, R. A. (2009). Cognitive and social help giving in online teaching: an exploratory study. Educational Technology Research and Development, 57(2), 169-192. doi:10.1007/s11423-008- 9104-7 Corresponding author: Josh McCarthy, josh.mccarthy@unisa.edu.au Australasian Journal of Educational Technology © 2015. Please cite as: McCarthy, J. (2015). Learning in the Café: Pilot testing the collaborative application for education in Facebook. Australasian Journal of Educational Technology, 31(1), 67-85. Introduction Case studies using Facebook as an e-learning environment E-learning with other social media tools Aim and research questions Designing the Café Table 1 The five principles of user interface design for the design of the Café Method Table 2 The breakdown of student demographics within the cohort Table 3 Student responses to questions and statements It's something new and different. It's very exciting. Utilising the Café Figure 3. Left: a screen capture of the forum pinboard; right: a screen capture of a forum gallery Student responses to the learning environment within the Café Table 5 Student responses to questions and statements from the post-semester survey Table 6 Student responses regarding online and in class interaction, from the post-semester survey Student performance within the course Figure 6. Student academic performance within Design Language in Media Arts Figure 7. Average number of visits to the Café per student grade bracket Figure 8. Average number of posts to the Café galleries per student grade bracket Figure 11. Average number of comments received from peers per post per student grade bracket Discussion References Vaughan, T. (2011). Multimedia: Making it work (8th ed.). New York, NY: McGraw-Hill.