ching.pdf


Australasian Journal of
Educational Technology

2011, 27(Special issue, 5), 781-797

Design-grounded assessment: A framework and a case
study of Web 2.0 practices in higher education

Yu-Hui Ching and Yu-Chang Hsu
Boise State University, USA

This paper synthesises three theoretical perspectives, including sociocultural theory,
distributed cognition, and situated cognition, into a framework to guide the design
and assessment of Web 2.0 practices in higher education. In addition, this paper
presents a case study of Web 2.0 practices. Thirty-seven online graduate students
participated in a small-group collaborative concept mapping activity using Web 2.0
applications (e.g. W e b s p i r a t i o n ) to construct sophisticated understanding of
instructional design processes. The analysis of this case focuses on different
assessment strategies adopted to ensure students’ successful participation in such
technology-rich collaborative context. This case study concludes that a shared goal
needs to be in place to establish a purpose of collaboration. The collaborative nature of
learning afforded by Web 2.0 applications needs to be acknowledged through the
award of grades. That is, both the processes and products of collaborative knowledge
construction need to be assessed and formally graded at individual and group levels.
This paper also suggests several potential assessment strategies that may enhance
smoother Web 2.0 practices, and discusses some possible challenges associated with
those strategies.

Introduction
The new generation of web-based technologies, termed Web 2.0 technologies by
O’Reilly (2005), enables easy publishing, sharing, and communication. Distinguished
from previous web technologies, Web 2.0 technologies, such as blogs, wikis, social
bookmarking tools, and collaborative concept mapping tools, allow content creators to
easily publish their work via the web without knowing HTML codes. In addition, the
innovative features of Web 2.0 technologies lie in their automatic dissemination,
powerful organisation, enhanced interactivity, and simplified collaboration.  Through
the Really Simple Syndication (RSS) feature, content shared via Web 2.0 technologies can
be automatically disseminated to RSS feeds subscribers (Kim, 2008). The tagging
function of Web 2.0 technologies empowers content creators to organise their creative
work by assigning key words attached to the specific piece of work, while helping
content consumers search for appropriate content using “tags” (Hsu, Ching &
Grabowski, 2009). In a sense, the web content is organised by and accessible through
tags. The communication features, including commenting and chatting functions,
provide venues for content consumers to interact and engage in conversation with
content creators. Certain Web 2.0 technologies provide real time collaborative editing
and recorded revision history features that help streamline the web-based
collaboration processes and potentially increase productivity.

Web 2.0 technologies have been adopted widely in higher education for teaching and
learning. The characteristics of easy publishing, sharing, and communication of these



782 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

technologies encourage and help create a participatory culture among educators and
learners (Hsu et al, 2009). This participatory culture values educational practices that
promote collaborative learning and peer-to-peer interaction (Gunawardena, Hermans,
Sanchez, Richmond, Bohley & Tuttle, 2009), that involve active participation and
interactive multi-way communication, and that engage learners in knowledge creation
activities (Dohn, 2009). Such educational practices enabled by Web 2.0 technologies,
namely Web 2.0 practices, help learners develop essential skills needed for success in
today’s world, such as critical thinking, problem solving, communication, and
collaboration.

Although higher education has been taking advantage of Web 2.0 applications to
create technologically-enriched learning experiences for students, most of the existing
Web 2.0 literature shows that educators did not use those applications to their full
potential. That is, the participatory, interactive, collaborative, and social aspects were
often missing from the learning (Ching & Hsu, 2010) and assessment activities (Gray,
Thompson, Sheard, Clerehan & Hamilton, 2010). In a critical review of blogging
activities in higher education, Ching and Hsu (2010) found that a large number of
blogging activities were mostly designed to facilitate or promote individual reflective
thinking or practice. Although interaction with peers through the commenting feature
was often encouraged or required, it was found that active interaction with peers in
blogging activities was not guaranteed (e.g., Ellison & Wu, 2008).

Since assessments usually drive learning, it is likely that the lack of active peer
interaction was a result of failing to include relevant criteria for assessing peer
interaction as part of the learning performance (Macdonald, 2003). In addition, the
interaction among students was often found to be at a superficial level because
students typically do not feel comfortable critiquing or they lack the necessary skills to
provide constructive feedback (e.g., Xie, Ke & Sharma, 2008; Wassell & Crouch, 2008;
Farmer, Yue & Brooks, 2008; Fessakis, Tatsis & Dimitracopoulou, 2008). Gray et al.
(2010) analysed cases in which students created content with Web 2.0 applications in
higher education settings. They found that in most cases, the grading criteria for Web
2.0 authoring still represent the more conventional modes of assessment that use
written work as the major, if not the sole basis for grading. They found that only in
some cases students’ uses of interactivity or social interaction features were assessed,
while students’ uses of innovative affordances provided by Web 2.0 tools, such as
tagging, RSS feeds, communication features, and revision history were rarely assessed.

The two aforementioned reviews of the existing Web 2.0 literature by Ching and Hsu
(2010) and Gray et al. (2010) identified the incongruence between how Web 2.0
applications were commonly used in education and the core value of the Web 2.0
practices. The identified incongruence could potentially decrease students’ motivation
in using the applications for social learning and knowledge creation. As such,
researchers have proposed the need for useful frameworks that can guide the
development of sound pedagogical strategies for integrating Web 2.0 applications as
well as assessment endeavors of Web 2.0 practices (Gray et al., 2010; McLoughlin &
Lee, 2010; Laurillard, 2009).

In this paper, we present a synthesised framework that can not only guide the design
of learning experiences afforded by Web 2.0 applications and aligned with core value
of Web 2.0 practices, but also serve as the foundation for developing assessments of
students’ Web 2.0 practices. We also analyse a case of Web 2.0 practices in higher



Ching and Hsu 783

education using the synthesised framework. Based on the case study, we provide
suggestions on assessment strategies that may enhance smoother Web 2.0 practices,
and discuss some possible challenges associated with those strategies.

Theoretical framework

Three theoretical frameworks were selected because of their relevance to the core value
of educational practices afforded by Web 2.0 technologies: 1) socio-cultural theory
(Vytgotsky, 1978); 2) distributed cognition (Pea, 1993; Bell & Winn, 2000); and 3) situated
cognition (Brown, Collins & Duguid, 1989; Lave, 1988).

Vygotsky's sociocultural theory (Vytgotsky, 1978) emphasises the critical role of social
environment in facilitating development and learning (Tudge & Scrimsher, 2003). The
social environment mainly refers to the interpersonal or social interactions. It enables
constructing meaningful knowledge and transforms learning experiences (Schunk,
2008). The social environment impacts individuals' learning through various tools,
including cultural objects (e.g., machines), language, and social institutions (e.g.,
schools). Among the tools, language is the most powerful one because it is the
mediator people use directly for interpersonal/social interactions. From the
sociocultural perspective, Web 2.0 applications are ideal for mediating social
interactions because they can not only serve as the repository of the mediation
processes and products (e.g., a wiki system hosts written drafts or VoiceThread hosts
audio comments), but also help social interactions extend beyond geographical and
temporal constraints.

While sociocultural theory puts emphasis on knowledge construction through social
interaction among human agents, the distributed cognition perspective extends the
agents of interactions beyond human beings. Distributed cognition posits that
knowledge is distributed across collaborators, external symbolic representations, tools,
environments, and artefacts (Pea, 1993; Bell & Winn, 2000). Considering the major
argument of distributed cognition, this theoretical perspective contributes to extending
the view of cognitive activities constrained in each individual's mind to any unit
external to one's mind, including other people, tools, and artefacts. According to this
theoretical perspective, the benefit of knowledge is maximised only when it is
represented externally for the utilisation of involved individuals. In terms of contexts,
distributed cognition occurs in two major types of settings—natural and by design
(Bell & Winn, 2000).

A good example of distributed cognition in a natural setting is the intellectual activities
during problem solving where distributed cognition exists in the individuals who
participate in problem solving activities. In the context of distributed cognition by
design, cognition is recorded on the tools when human beings interact with the tools
that are designed to support work or tasks, such as devices (e.g., calculators) or
software (e.g., Web 2.0 applications). Accordingly, a calculator contains distributed
cognition during calculation tasks, and Web 2.0 applications (e.g., wikis) record both
processes and products of knowledge construction. Bell and Winn (2000) indicated
that highly interactive and networked media motivated the pursuit of a distributed
cognition perspective, because this perspective is especially useful in explaining
meaningful interaction and collaboration at a distance. Web 2.0 applications, as highly
interactive and networked media, serve as ideal tools that enable meaningful
interaction and collaboration at a distance. As cognitive tools, these applications help



784 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

relieve cognitive burden by externalising one's cognition and making it visible to
collaborators. As enablers of collaboration, Web 2.0 applications provide platforms for
knowledge creation and recording, which makes both the collaboration process (e.g.,
tracking revision history on a wiki) and products (e.g., working document) visible not
only to all collaborators but also in almost real time. Furthermore, cloud-based and
networked Web 2.0 applications afford updating work-in-progress instantly for
effective collaboration at a distance. For example, collaborative writing/editing
applications, such as Google Docs, allow multiple people to edit a shared document at
the same time. Collaborators do not need to send the document back and forth via
emails for collaboration. The visibility of collaborative process and updated level of
collaborative work reduce the wait time and increase efficiency in communication.

Situated cognition focuses on the context of the interaction for knowledge construction,
which complements the two theoretical views above. Situated cognition emphasises
learning and practices in authentic and meaningful contexts (Greeno et al., 1998;
Brown, Collins & Duguid, 1989; Lave, 1988). The contribution of this perspective
toward explaining Web 2.0 practices resides in the emphasis on learning in context
(e.g., activity, people, culture, and language) and learning is inseparable from doing
(Brown, Collins & Duguid, 1989). When individuals construct knowledge together,
they reciprocally create learning experiences for each other, and serve as part of each
other’s "learning environment." Situated cognition also draws from the perspective of
community of practice where a group of people with common interests develop and
evolve together (Lave & Wenger, 1991). Through sharing knowledge, experiences, and
practice, a community of practice develops knowledge related to their field (Lave &
Wenger, 1991). With the built-in affordances for communication and collaboration,
Web 2.0 applications provide and serve as the environments to not only make learning
by doing (participating) possible, but also allow for learning by “participating
together", which enhances the richness and opportunities of learning potential and
experiences. In communities of practice, knowledge and competence are situated
cognition resulting from situated participation (Dohn, 2009).

Synthesising the three theoretical perspectives, Web 2.0 practices value knowledge
construction through social interactions (Gunawardena et al., 2009), view knowledge
as distributed among involved entities (Bell & Winn, 2000), and embrace learning
through situated participation (Dohn, 2009). As such, Web 2.0 practices would engage
learners in representing and organising their knowledge for knowledge construction,
and in actively interacting with other people and available tools in an authentic and
meaningful environment. Web 2.0 applications can be powerful tools with great
potential to enhance Web 2.0 practices: 1) Web 2.0 applications in themselves not only
support social and interpersonal interactions through their interactive affordance/
functionality (e.g., chat and commenting function), but also support the use of a
powerful mediation tool - language in various formats (text/video/audio); 2) Web 2.0
applications can be hosts of the distributed cognition of collaborative individuals and
groups by recording the externalised cognition; 3) Web 2.0 applications provide
environments to build authentic learning contexts in which collaborators engage in
collaborative knowledge construction through situated participation.

Despite the powerful affordance of Web 2.0 applications, Web 2.0 practices will not
simply “happen” if there is no deliberate design of the learning and assessment to
enable the practices. To promote Web 2.0 practices, several aspects should be
addressed in any design and assessment of learning activities, including shared goals,



Ching and Hsu 785

interaction for knowledge construction, and process and product artefacts. A shared
goal (e.g., a common task to complete or a common problem to solve) establishes a
purpose for interaction among the members of a learning group or community. A
shared goal, as the glue, bonds the members to work together - whether it is a small
group formed in a biology class or an informal learning community aggregated to help
each other on questions regarding mobile programming. Without shared goals,
participation, communication, and collaboration are unlikely to exist. When promoting
Web 2.0 practices for learning, explicitly providing a shared goal to learners or
facilitating the creation of a shared goal by learners, will help establish the foundation
of successful collaborative work. However, to understand whether Web 2.0 practices
are successful, it is important to assess if the interaction, the process, and product
artefacts reflect and address the shared goal of a group or community. Fortunately, the
features of Web 2.0 applications provide the possible tools to help with what is needed
for such assessment. These applications often can record interaction among
collaborators (e.g., comments or conversation on the work), collaboration process (e.g.,
tracking revision history) and produced artefacts (e.g., jointly written drafts). These
records are the cognitive residue or "tangible distributed cognition" of learning groups
or communities, providing rich data for assessing the participation, communication,
and collaborative knowledge construction toward achieving the shared goals. Figure 1
presents a synthesised graphic representation of the previously reviewed frameworks.

Figure 1: A synthesised framework for Web 2.0 activity design and assessment

Design and assessment considerations

During the process of Web 2.0 practices, the shared goals, collaborative process,
artefacts produced, and affordances of Web 2.0 applications are all interdependent and
complement each other to help achieve social learning in contexts that are authentic
and meaningful to the groups and communities. Deliberate efforts are required to
design and assess Web 2.0 practices for creating complex yet effective interactions
among the various elements mentioned above. We generated the following questions
to help guide the design and assessment of Web 2.0 practices, and categorised the
questions by the major components of the framework: shared goals, knowledge
construction through social interaction, process and product artefacts, and authentic
and meaningful contexts.



786 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

Shared goals:
• What are the explicit shared goals of the groups?
• What are the implicit/implied/induced shared goals of informal learning

communities or communities of practice?
Sociocultural theory: Knowledge construction through social interaction

• How do we design a structure of interaction that engages learners in discussing
and negotiating meaning to help achieve the shared goals?

• What are the Web 2.0 applications available in this social environment to
support meaning negotiation during the learning process?

Distributed cognition: Process and product artefacts
• What are expected and considered as tangible representations of distributed

cognition, including process and product artefacts?
• Are those tangible representations aligned with the shared goals of learning

groups and communities?
Situated cognition: Authentic and meaningful contexts

• Does learning occur in authentic contexts that require participation,
communication, and collaboration, and that are meaningful to the
groups/communities of learners?

The case study: Designing and assessing Web 2.0 practices

Method

In this section, a case of Web 2.0 practices in higher education is presented followed by
an analysis of the case using the aforementioned framework. This case study was
conducted in the context of an online master’s level course in a northwestern state
university in the United States. The subject matter of this course is Instructional Design,
one of the core courses students are required to take in this Educational Technology
master’s degree program. Thirty-seven learners were enrolled in two sections of this
course, taught by the first author. Many of these learners were K-12 school teachers,
with some college instructors, technology coordinators, technical writers, and
instructional designers in corporate settings. These learners were geographically
distributed, most of them living in different states in the U.S., one in Canada, one in
Asia, and one in Europe.

The Web 2.0 activity reported in this case study was a collaborative concept mapping
activity, one of the integrated learning activities for this online graduate course. Thirty-
seven students participated in this Web 2.0 activity, as they would normally do with
other course activities. All forms of data for this study were retrieved after the course
had ended to ensure there was no bias in assigning students’ course grades. All
students’ identifiers were removed so that the data could not be associated with
individual student. The artefacts generated during this activity process were collected,
including group concept maps, concept map revision history frequency counts, and
chat logs. In addition, an open-ended question inquired about students’ perceptions of
the concept map activity in an anonymous mid-term feedback survey as part of the
course evaluation. At the end of the Web 2.0 activity, students were given an
opportunity to assess self and peer performance. Students were asked to comment on
their group collaboration process and how the design of the activity could be
improved in two open-ended questions. Qualitative data were examined for themes
relevant to the synthesised framework.



Ching and Hsu 787

The design of the collaborative concept mapping activity

In this introductory instructional design course, learners were introduced to the
ADDIE (Analysis, Design, Development, Implementation, Evaluation) process model
and were expected to study several other instructional design models relevant to their
professional contexts. Because collaboration is deemed a critical competence of
instructional design (Richey, Fields & Foxon, 2001; Ritzhaupt, Martin & Daniels, 2010),
students were provided with abundant opportunities to develop their communication
and collaboration skills. Furthermore, one of the course goals was to acquaint learners
with Web 2.0 applications and to help them explore the potential of these applications
for their own teaching or professional practices. Thus, Web 2.0 applications were
integrated into different course activities to support productivity, communication, and
collaboration for the coursework.

The Web 2.0 activity for this case study aimed at having learners collaboratively create
sophisticated graphic representations of the instructional design process that could
guide their future instructional design endeavour. Prior to the activity, learners self-
selected themselves into groups of three or four. The activity involved the following
steps: 1) learners individually chose an existing instructional design model and
presented it in a concept map format (Novak & Cañas, 2008); 2) learners shared their
individually created concept map with their group members on a Web 2.0 concept
mapping platform (i.e., Webspiration); 3) learners collectively integrated several
different selected models into a coherent representation of an instructional design
process. Scaffolding guidelines were provided to achieve the group shared goal of
creating an integrated concept map. Learners were encouraged to discuss the
similarities and differences of each model, the relevance of each model to the ADDIE
process, and the presentation of an integrated concept map in a comprehensive and
coherent way. The entire learning activity lasted for one and a half months, including
two submissions of the group concept maps. The rationale for such an extended period
was to accommodate scheduling of the group work due to the geographical separation
of these online students. Students were encouraged to utilise different channels for
communication with their group members during the process, including a discussion
forum set up on Moodle (the course management system used for this course), emails
via university accounts, and other tools of individual group’s choice. This group
activity accounted for 10 percent of the total course grade.

Knowledge representation using Web 2.0 applications

The collaborative concept mapping activity itself serves as a learning and assessment
activity. A concept map is a type of external graphical representation for organising
and representing knowledge (Novak & Cañas, 2008). The active construction of
concept maps is suggested to engage learners in deeper cognitive processing, such as
externalisation, explanation, and elaboration, resulting in learners refining their
understanding of the complex subject matter (Ainsworth, 2006; Janssen, Erkens,
Kirschner & Kanselaar, 2010). When concept maps are constructed by a group of
learners and supported with proper technologies, learners are more likely to
externalise their thinking for peers to view, provide clarifications, and participate in
discussions that lead to joint understanding of the subject matter (Janssen, et al., 2010).
Concept maps can also be used as a tool for evaluating individual’s understanding of
the subject matter (Novak & Cañas, 2008). The format of concept maps exemplifies: 1)
nodes that represent concepts, and 2) lines and arrows linking two concepts,
representing the relationships between the concepts. As individuals and groups gain



788 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

more expertise in an area, the created concept maps will show increased numbers of
nodes and links, representing their more sophisticated understanding of the subject
matter. Because collaborative concept mapping involves several individuals, group
concept maps are expected to be more sophisticated and comprehensive than an
individually created concept map.

The assessments of the activity

A variety of strategies were adopted to assess students’ learning in this collaborative
concept mapping activity enhanced by the Web 2.0 application, including: 1) formative
feedback on knowledge construction and communication; and 2) summative
assessments on group concept maps and self as well as 3) peer assessments on one’s
contribution to the collaborative process. Overall, these strategies assessed both
student cognitive learning and collaboration at group levels.

Formative feedback

Group concept maps. Group concept maps were submitted first for formative feedback
through which the instructor provided constructive feedback to help learners revise
their group work for final submission. The rubric for assessing the integrated group
concept maps includes the following criteria:

1. Whether the integrated map presents a sophisticated understanding of the
instructional design process by including important concepts and relations between
the concepts;

2. How well the included concepts were associated with the ADDIE process model;
3. Whether the concept map follows the required format that nodes represent

concepts and links denote relationships between concepts;
4. Whether the integrated map is concise and coherent (i.e., no redundant concepts).

The instructor provided detailed comments on group concept map for each group,
praised group collaborative efforts, outlined areas for improvement, and encouraged
group revision based on the feedback. No grades were assigned to any group members
at this time.

Communication. The instructor monitored the collaborative efforts by examining the
group discussion forums and sent email reminders to groups that lacked
communication. It was observed that learners utilised a range of communication
channels to share and discuss ideas, and to manage group work. In addition to the
instructor-created discussion forums, several groups communicated synchronously or
asynchronously using the chat function embedded in the recommended Web 2.0
collaborative concept mapping application. When group members logged on to the
collaboration platform at the same time, they chatted synchronously. As the chat logs
were preserved by the system automatically, members were able to view or review the
logs afterwards. In addition, one group used the “Commenting” function embedded in
the application. Learners also reported using emails to communicate and they
sometimes copied the instructor on the emails to update their collaboration process or
to ask instructor assistance for solving group collaboration issues. Although learners
also used video conferencing tools for synchronous discussions, the instructor had no
access to student communication via these channels. Through accessing discussion
boards, chat logs, and student email exchanges, the instructor was able to monitor the
collaboration process and helped resolve group collaboration issues. For example, the



Ching and Hsu 789

instructor mediated conflicts between two members in a group and fostered an
understanding of different working styles.

Summative assessments

Group concept maps. After the groups received formative feedback on how to improve
their group maps, they were given another two weeks to revise and submit final group
maps for grading. Group concept maps were assessed at the end of the activity based
on the rubric presented in the Formative feedback section above. Overall, eight out of ten
groups performed well and achieved the learning goals of the activity with three of
these groups performing exceptionally well. These top-performing groups created
well-organised concept maps with sufficient sophistication. Their maps covered a
great deal of the important concepts by including 65 to 75 nodes. These concepts were
also well connected to each other, including 75 to 80 links. Figure 2 presents an
example of the well-organised and integrated group concept map. On the other hand,
the two groups with low-quality concept maps included fewer than 35 nodes each —
one group did not have all the nodes linked properly and the other submitted a poorly
organised concept map with redundant nodes and arbitrary links. Figure 3 shows an
example of a poorly organised and integrated map.

Figure 2: A well organised and integrated group concept map

Self and peer assessments. After learners submitted their group maps for grading, they
were asked to fill out an online survey of self and peer assessments. In the survey,
learners were asked to assess their own contribution as well as the contributions of all
of their teammates during the collaborative process. They were told that their
responses would be confidential and would not be shared with their peers. On a 5-
point Likert scale, learners were asked to rate themselves and their peers on four
questions: 1) the quality of their contributions to the group; 2) a fair share of
contribution to the group work; 3) cooperation and communication with other group
members; and 4) cognitive contributions on helping the group accomplish its goals.
Question 1 and 4 served as indicators concerning the quality of cognitive contribution,



790 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

whereas Question 2 and 3 served as evidences of involvement and cooperation during
the group process. Two open-ended questions asked learners about their perception of
learning during the collaborative activity and how the design of the activity could be
improved for smoother collaboration.

Figure 3: A poorly organised and integrated group concept map

Examining the results of self and peer assessments, we found that in the well
functioning groups, learners were rated similarly by themselves and by their group
members and they were mostly rated positively at 4 or 5 on a five-point Likert scale for
each of the four aforementioned questions. In one case where there was a non-
participating member, all the other members in that group did not include this non-
participating member in their peer-assessments, which implied the non-participating
member was not considered as part of the group by his peers. In some cases,
disagreement was apparent in the peer-assessments, indicating group members had
different perceptions about their involvement and contribution to the group work.
This occurred in one group where two members did not get along due to their coming
from different professional backgrounds and one person tried to dominate the group
process while the other fought for her voice. This situation caused the group to suffer
from an unsmooth group process, as evidenced in another member’s comments in her
responses to the open-ended question.

Overall, after triangulating among several data sources, the results of the self and peer
assessments showed that this assessment strategy could reveal the group process and
interaction quality. That is, the high consistency between self assessments and peer
assessments could indicate the unproblematic group interaction and collaborative
process; whereas inconsistency in these assessments could reveal possible conflicts or
problematic group involvement (Griesbaum & Gortz, 2010).

Discussion: What we learned from assessing Web 2.0 practices

The Web 2.0 practices examined in this case were situated in a collaborative concept
mapping activity supported by a Web 2.0 application. To help learners construct
subject matter knowledge of instructional design, they were asked to represent their



Ching and Hsu 791

knowledge individually, discuss and negotiate their understanding within their small
group, and construct knowledge collaboratively to demonstrate their group
understanding of the subject matter. The collaborative activity also provided
opportunities for learners to develop their communication and collaboration skills. The
multiple layers of the activities, including representing and constructing knowledge at
both individual and group levels, require a set of strategies to assess the multiple
intended goals. In this section, applying the synthesised framework, we discussed how
we assessed shared goals, process and product artefacts, and social interaction in the
Web 2.0 activity. We also suggest possible assessment strategies for future
implementations and discuss the challenges associated with assessing Web 2.0
practices.

Assessments targeted at shared goals

In this case study, the instructor created a shared goal for learners to promote
interaction for knowledge construction and to create an authentic context for
communication and collaboration. The grading policy stressed that learners were
mainly evaluated based on their group products, except for the cases where
contributions from certain individuals were seriously lacking. We found that groups
adopted multiple communication channels to interact and collaborate during the Web
2.0 activity, suggesting that the need to achieve a shared goal creates a meaningful
context that promotes interaction and collaboration among members. Overall, the
learning outcomes showed that most groups were able to achieve the shared goal well.
Namely, they produced group concept maps representing sophisticated group
understanding of instructional design processes. By examining the qualitative data on
learner perceptions of the group activity, we found that learners who enjoyed the
challenge of achieving a shared goal had the following comments:

It was nice to have a group to work with because much of what we are learning is new
information and it is always nice to bounce ideas off of someone else.

By seeing how other’s interpreted the [instructional design] models I was able to
understand them better than just reading from the coursework.

Yet another student started to embrace the activity toward the end, stating that:

I felt this activity was a fantastic learning experience, not just for instructional design
but for collaboration. Our group actually fought a little bit but we worked through it
and came out better for it in the end.

However, some students did not value the collaborative knowledge construction
aspect of the activity that is essential to achieve the shared goal. They commented on
their preference for working individually, especially in an online learning environment
where face to face interaction is lacking. Students in this course were mostly busy
professionals with full time jobs and families to attend to. To coordinate a group of
three to four members can be rather challenging, especially when members are located
in different time zones. A few students commented:

I would scrap this activity in favor of individual work with a group comment option. It
was quite challenging to collaborate on this document given the asynchronous nature of
the course.

The group process is very difficult in this learning environment. For the amount of time
we spent, I could have very easily created my own blended model without a group.



792 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

Moreover, some believed they did not learn from participating in the group work. A
student commented that the group work:

... caused nothing but unwanted stress and frustration and wasted a great deal of
time… since I couldn’t get any group members to respond.

In addition, some students did not feel comfortable being graded solely on the basis of
their group work. A student indicated that his/her grade relied on people he/she had
not met before. In some cases, learners were frustrated by their group mates who were
not motivated to achieve the shared goal and displayed irresponsible behaviors, such
as completing work at the last minute or simply failing to complete their own share of
the work. These behaviours caused potential conflicts in the group collaboration
process, which exacerbates the discomfort of being graded solely for group product. It
was also found that learners in the well-functioning group generally showed positive
attitudes toward this activity and believed that they learned the subject matter better
while engaging in this activity. On the other hand, members in dysfunctional groups
tended not to value group activity and preferred to work on their own, possibly due to
either having irresponsible group members or having conflicts among members.

The balance between shared goals and individual learning

In this case study, grading on the achievement of the shared goal holds learners
accountable at the group level. On reflection, although a shared goal could provide a
context and help establish a purpose for communication and collaboration, individual
learning should also be reinforced by holding each learner accountable in their Web
2.0 practices. Based on previous research findings that individual accountability tends
to increase achievement in computer-supported collaborative learning, Johnson and
Johnson (2008) suggested individual accountability should be built into a collaborative
activity by assessing the performance of each member. As much as assessments drive
student learning, a Web 2.0 activity needs to ensure that both the individual learning
and shared goals are assessed. For example, the reported collaborative concept
mapping activity could have required learners to create an individual concept map
that accounts for a portion of the total activity score. Such an individual task should
have a clear due date coming before the beginning of the group effort. Instructors can
take this opportunity to interact with individuals and assess the individual
preparation for the collaborative activity.

This way, the activity grade would not depend solely on the group work and
individual’s work could be valued by a knowledgeable authority (i.e., the instructor).
Individuals might, therefore, feel they have better control over the collaborative
activity and become more receptive and less intimidated by the shared goal. However,
previous research also identified possible tensions between assessing individual
endeavour and group products because students would prefer more weighting on
individual contribution than on group products (e.g., Macdonald, 2003). In addition, a
limitation of assessing at both individual and group levels lies in increasing the
instructor’s workload. Depending on the load of the activity in the context of the
overall course design, grading and providing feedback to every individual for the
purpose of ensuring individual accountability may not be feasible in a class with large
enrolment (Macdonald, 2003).



Ching and Hsu 793

Assessing process as well as product artefacts

Web 2.0 practices facilitate learning through participation, interaction, and
communication, and place value on the learning processes where social interaction and
meaning negotiation take place. In addition, the use of Web 2.0 applications helps
capture the cognitive and collaborative process during the activity, allowing early
diagnoses of learner’s understanding of the subject matter and potential issues of
group processes. In the case presented above, concept mapping was used as a strategy
for learners to represent their individual knowledge and for jointly constructing group
knowledge that reflects the more sophisticated group understanding of the subject
matter. Taking the distributed cognition perspective, during this Web 2.0 practice,
knowledge is distributed among the group members, the Web 2.0 collaborative
platform, other communication tools (e.g., discussion forums and emails), and the
process and product artefacts (e.g., individual concept maps, group concept maps,
communication logs, and comments). Assessing both learning processes and products
in this study helped the instructor and researcher gain a holistic picture of the intricate
cognitive and collaborative activities occurring in the Web 2.0 practice.

Using Web 2.0 applications has the conspicuous advantage of enabling assessment of
both processes and products. In this case study, learner knowledge representation,
knowledge construction, and interaction were recorded by the Web 2.0 application and
could be viewed and reflected upon at a later time. Through the recorded interactive
activities, such as chat logs, or asynchronous messages posted on the discussion
boards, the process of collaboration was made more transparent. These process data
can not only serve as sources for examining collaborative processes and individuals’
contribution to those processes (Macdonald, 2003), but also help learners reflect on the
evolution of their thinking while engaging in Web 2.0 practices. In this study, the
instructor provided formative feedback on group knowledge construction during the
collaborative process and allowed for another submission after group revision based
on the feedback. Some learners indicated that such structure helped relieve their stress
during the group process because they received an opportunity for revision and
directions for improvement. In addition, examining learner interaction through
recorded interaction patterns helped shed light on the collaboration process. Assessing
such processes enables the diagnosis of potential participation issues and allows for
early intervention.

Assessing social interaction and collaboration

In this study, several data sources were used to help assess social interaction and
collaboration. The results of self and peer assessments on individuals’ contribution to
the collaborative process helped detect positive and negative effects of collaborative
learning, and revealed “a picture of perceived interaction quality” (Griesbaum &
Gortz, 2010, p.484). In addition, having learners assess their own contribution helps
learners take control over their learning (McConnell, 1999), and reflect on their
interactive as well as collaborative skills.

The revision history recorded automatically by the Web 2.0 application also disclosed
the collaboration process through recorded edits and time stamp for each edit. We
found that groups with quality final products tended to have higher frequency of
revisions. The groups that did not collaborate smoothly and submitted less than
satisfying final group products generated about one third to half of the revision



794 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

frequency compared to those of the high performing groups. The revision history also
showed how each member contributed during the collaborative process. An extreme
example in this study showed that a member in a group only added his portion to the
group product after the rest of the group had finished their collaboration, resulting in a
poorly integrated final group concept map. Web 2.0 applications as collaborative
platforms offer group members equal access to the created artefacts, hence allowing
individuals to have their voice during the group process and reducing the possibility
of a dominant member controlling the whole process (Vratulis & Dobson, 2008).
However, in this example, equal access seems to present new challenges for managing
the group process. In addition, using the revision history as a data source has its
limitations. For example, a group may work on more than one document or on
different collaborative platforms throughout the process, which makes it difficult to
track the revision history. As such, revision history may not always be comprehensive
or reliable as a data source for assessment.

The communication logs (e.g., chat logs) could also shed light on group work
processes and help detect disengaged members as well as potential conflicts that
occurred during the collaboration. However, as learners in this case study utilised a
range of tools to discuss and manage their group work, the communication became
dispersed among different tools. This also presented a challenging situation for the
instructor to track all the communication that occurred. The dilemma, thus, lies
between offering learner choices of communication tools that allow autonomy and
satisfy diverse learner needs, and prescribing certain tools for easier management. One
possible solution could be allowing learners to use their preferred means of
communication but requiring them to submit their communication logs that
demonstrated their group effort and individual contribution.

Conclusion

Web 2.0 applications have taken their roots in higher education today, but these
applications have not been used to their full potential to achieve real Web 2.0
practices—to value the participatory culture, to emphasise collaborative learning and
peer to peer interaction, to promote active participation and interactive multi-way
communication, and to engage learners in knowledge creation activities. In this paper,
we proposed a synthesised framework to help guide the design of assessment of Web
2.0 practices in higher education. We also presented a case to illustrate how we
assessed shared goals, process and product artefacts and social interaction and
collaboration in a graduate online course. This case study suggests that a shared goal
that establishes a purpose for collaboration needs to be assessed to ensure interaction.

In addition, the collaborative nature of learning afforded by Web 2.0 applications
needs to be acknowledged through the award of grades. That is, both the processes
and products of collaborative knowledge construction need to be assessed and
formally graded at individual and group levels. While the focus of this paper is on
designing assessments on Web 2.0 practices in higher education, we believe the
activity and assessment of Web 2.0 practices are inseparable parts of quality learning
experiences. The results of assessments could help inform and improve the next
iteration of learning activity design. We hope the proposed framework, assessment
design considerations, and the presented case study can together inform educators
about some possible ways of design-grounded assessment of Web 2.0 practices in
higher education.



Ching and Hsu 795

Acknowledgments

The authors would like to thank Dr Ross Perkins for designing the early version of the
collaborative concept mapping activity.

References
Ainsworth, S. (2006). DeFT. A conceptual framework for considering learning with multiple

representations. Learning and Instruction, 16(3), 183-198.
http://dx.doi.org/10.1016/j.learninstruc.2006.03.001

Bell, P. & Winn, W. (2000). Distributed cognitions, by nature and by design. In D. H. Jonassen &
S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 123-144). Mahwah, NJ:
Lawrence Erlbaum Associates.

Brown, J. S., Collins, A. & Duguid, P. (1989). Situated cognition and the culture of learning.
Educational Researcher, 18(1), 32-42.

Ching, Y.-H. & Hsu, Y.-C. (2010). Blogging in higher education: Issues, challenges, and design
considerations. Proceedings of Association for Educational Communications and Technology
(AECT) Annual International Convention, Anaheim, California. October 27-30, 2010.

Dohn, N. B. (2009). Web 2.0: Inherent tensions and evident challenges for education. International
Journal of Computer-Supported Collaborative Learning, 4(3), 343-363.
http://dx.doi.org/10.1007/s11412-009-9066-8

Ellison, N. & Wu, Y. (2008). Blogging in the classroom: A preliminary exploration of student
attitudes and impact on comprehension. Journal of Educational Multimedia and Hypermedia,
17(1), 99-122. http://www.editlib.org/p/24310

Farmer, B., Yue, A. & Brooks, C. (2008). Using blogging for higher order learning in large cohort
university teaching: A case study. Australasian Journal of Educational Technology, 24(2), 123-
136. http://www.ascilite.org.au/ajet/ajet24/farmer.html

Fessakis, G., Tatsis, K. & Dimitracopoulou, A. (2008). Supporting “Learning by design” activities
using group blogs. Journal of Educational Technology & Society, 11, 199-212.
http://www.ifets.info/journals/11_4/15.pdf

Gray, K., Thompson, C., Sheard, J., Clerehan, R. & Hamilton, M. (2010). Students as Web 2.0
authors: Implications for assessment design and conduct. Australasian Journal of Educational
Technology, 26(1), 105-122. http://www.ascilite.org.au/ajet/ajet26/gray.html

Greeno, J. G. & the Middle School Mathematics Through Applications Projects Group. (1998).
The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26.
http://www.sciencedirect.com/science/article/pii/S0003066X02008687

Griesbaum, J. & Gortz, M. (2010). Using feedback to enhance collaborative learning: an
exploratory study concerning the added value of self- and peer-assessment by first-year
students in a blended learning lecture. International Journal on E-learning, 9(4), 481-503.
http://www.editlib.org/p/32313

Gunawardena, C. N., Hermans, M. B., Sanchez, D., Richmond, C., Bohley, M. & Tuttle, R. (2009).
A theoretical framework for building online communities of practice with social networking
tools. Educational Media International, 46(1), 3-16.
http://dx.doi.org/10.1080/09523980802588626



796 Australasian Journal of Educational Technology, 2011, 27(Special issue, 5)

Hsu, Y.-C., Ching, Y.-H. & Grabowski, B. (2009). The spirit of educational Web 2.0 literacy:
Cognitive tools of the new media age for K-12. In W. H. L. Tan & R. Subramaniam (Eds.),
Handbook of research on new media literacy at the K-12 level: Issues and challenges (pp. 353-371).
Hershey, PA: IGI Global.

Janssen, J., Erkens, G., Kirschner, P. A. & Kanselaar, G. (2010). Effects of representational
guidance during computer-supported collaborative learning. Instructional Science, 38(1), 59-
88. [verified 14 Aug 2011] http://igitur-archive.library.uu.nl/fss/2010-0115-
200338/Representational%20Guidance%20METIS.pdf

Johnson, D. W. & Johnson, R. T. (2008). Cooperation and the use of technology. In J. M. Spector,
M. D. Merrill, J. J. G. van Merriënboer & M. P. Driscoll (Eds.), Handbook of research on
educational communications and technology (3rd ed.) (pp. 401-423). Lawrence Erlbaum
Associates.

Kim, H. N. (2008). The phenomenon of blogs and theoretical model of blog use in educational
contexts. Computers & Education, 51(3), 1342-1352.
http://dx.doi.org/10.1016/j.compedu.2007.12.005

Laurillard, D. (2009). The pedagogical challenges to collaborative technologies. International
Journal of Computer-Supported Collaborative Learning, 4(1), 5-20.
http://dx.doi.org/10.1007/s11412-008-9056-2

Lave, J. (1988). Cognition in practice. Cambridge, England: Cambridge University Press.

Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK:
Cambridge University Press.

Macdonald, J. (2003). Assessing online collaborative learning: Process and product. Computers &
Education, 40(4), 377-391. http://dx.doi.org/10.1016/S0360-1315(02)00168-9

McConnell, D. (1999). Examining a collaborative assessment process in networked lifelong
learning. Journal of Computer Assisted Learning, 15(3), 232-243.
http://dx.doi.org/10.1046/j.1365-2729.1999.153097.x

McLoughlin, C. & Lee, M. J. W. (2010). Personalised and self-regulated learning in the Web 2.0
era: International exemplars of innovative pedagogy using social software. Australasian
Journal of Educational Technology, 26(1), 28-43.
http://www.ascilite.org.au/ajet/ajet26/mcloughlin.html

Novak, J. D. & Cañas, A. J. (2006). The theory underlying concept maps and how to construct and use
them. Technical Report IHMC CmapTools. Florida Institute for Human and Machine
Cognition. [verified 14 Aug 2011] http://cmap.ihmc.us/publications/researchpapers/
theorycmaps/theoryunderlyingconceptmaps.htm

O’Reilly, T. (2005). What is Web 2.0: Design patterns and business models for the next
generation of software. [verified 14 Aug 2011] http://www.oreillynet.com/lpt/a/6228

Pea, R. D. (1993). Practices of distributed intelligence and design for education. In G. Salomon
(Ed.), Distributed cognition: Psychological and educational considerations (pp. 47-86). Cambridge,
MA: Cambridge University Press.

Richey, R. C., Fields, D. C. & Foxon, M. (2001). Instructional designer competencies: The standards.
Syracuse, NY: ERIC.

Ritzhaupt, A., Martin, F. & Daniels, K. (2010). Multimedia competencies for an educational
technologist: A survey of professionals and job announcement analysis. Journal of Educational
Multimedia and Hypermedia, 19(4), 421-449. [verified 14 Aug 2011]
http://www.florencemartin.net/site08/research/MultimediaCompetencies.pdf



Ching and Hsu 797

Schunk, D. H. (2008). Learning theories: An educational perspective (5th ed.). Upper Saddle River,
NJ: Pearson.

Tudge, J. R. H. & Scrimsher, S. (2003). Lev S. Vygotsky on education: A cultural-historical,
interpersonal, and individual approach to development. In B. J. Zimmerman & D. H. Schunk
(Eds.), Educational psychology: A century of contributions (pp. 207-228). Mahwah, NJ: Erbaum.

Vratulis, V. & Dobson, T. M. (2008). Social negotiations in a Wiki environment: A case study
with pre-service teachers. Educational Media International, 45(4), 285-294.
http://dx.doi.org/10.1080/09523980802571531

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge.
MA: Harvard University Press.

Wassell, B. & Crouch, C. (2008). Fostering connections between multicultural education and
technology: Incorporating weblogs into preservice teacher education. Journal of Technology
and Teacher Education, 16, 211-232. http://www.editlib.org/p/22907

Xie, Y., Ke, F. & Sharma, P. (2008). The effect of peer feedback for blogging on college students’
reflective learning processes. The Internet and Higher Education, 11(1), 18-25.
http://dx.doi.org/10.1016/j.iheduc.2007.11.001

Authors: Dr Yu-Hui Ching PhD, Visiting Assistant Professor
Department of Educational Technology, Boise State University
1910 University Drive, Boise, Idaho, 83725-1747, USA
Email: yu-huiching@boisestate.edu

Dr Yu-Chang Hsu PhD, Assistant Professor
Department of Educational Technology, Boise State University
1910 University Drive, Boise, Idaho, 83725-1747, USA
Email: hsu@boisestate.edu

Please cite as: Ching, Y.-H. & Hsu, Y.-C. (2011). Design-grounded assessment: A
framework and a case study of Web 2.0 practices in higher education. In J. Waycott &
J. Sheard (Eds), Assessing students' Web 2.0 activities in higher education. Australasian
Journal of Educational Technology, 27(Special issue, 5), 781-797.
http://www.ascilite.org.au/ajet/ajet27/ching.html