Microsoft Word - [2]38-84-7-CE[2].docx Australasian Journal of Educational Technology, 2013, 29(6). ascilite 777 The competencies required for effective performance in a university e-learning environment Mitchell Parkes, Christine Reading University of New England, Australia Sarah Stein University of Otago, New Zealand The aim of this study was to identify and rate the importance of the competencies required by students for effective performance in a university e-learning environment mediated by a learning management system. Two expert panels identified 58 e-learning competencies considered to be essential for e-learning. Of these competencies, 22 were related to the use of technology. The remaining 36 competencies encapsulated a range of practices considered to be essential for learning within a social constructivist framework. Six of the competencies identified were either new or substantially different from what had been previously identified in the literature. A survey of e-learning stakeholders rating the importance of the e-learning competencies indicated that the competencies were not of equal importance. Critically, a number of key competencies from a social constructivist perspective that dealt with interacting and working with others were rated as being unimportant. This suggests that there is a disconnect between what the literature says about the importance of social constructivism to e-learning environments in theory and what e- learning stakeholders perceive its importance to be in practice. Introduction This paper reports on a study to identify the competencies required by students for effective performance in a university e-learning environment. The study was implemented in two phases. In phase one, the Hybrid Behaviourally Anchored Rating scale (Hybrid BARS) methodology was used to identify the competencies considered essential for e-learning. In phase two, the identified e-learning competencies were rated according to importance by a survey of e-learning stakeholders. The motivation for the study was twofold. First, to identify a set of e-learning competencies that could inform learning design and second, to use these competencies as the basis for training and development programs to help ensure stakeholders receive the full advantages of e-learning. The nature of competencies The term competency has been defined in the literature from a range of perspectives (Hoffmann, 1999). Having had application in a variety of areas including Education, Psychology, Management Theory, and Human Resource Management (Hoffmann, 1999), there is no precise or widely accepted definition for the term (de la Teja & Spannaus, 2008). Those who have used the concept of competency (or its plural competencies) have simply defined it in the manner that best suits the focus of their work (Hoffmann, 1999). Competencies are distinct from traits and characteristics. Traits and characteristics are personality descriptors typically formed early in life or inherited (Parry, 1998). Examples include "initiative", "self- esteem" and "decisiveness" (Parry, 1998, p. 60). Traits and characteristics resist change and are difficult to develop and measure (Cocanougher & Ivancevich, 1978; Parry, 1998). Competencies however, are clusters of related knowledge, skills and abilities that correlate with effective performance in the task or role at hand. Critically, competencies are measurable and can be developed and improved (Parry, 1998). E-learning competencies Effective performance in e-learning environments has been shown to require proficiency in a range of domains. Competencies identified in the literature considered critical for e-learning include: • student autonomy (Mayes & de Freitas, 2007; Stephenson, 2001; TESEP, 2007); • self-direction (Alexander & Boud, 2001; Bauer, Chin & Chang, 2000; Hedberg, 2001); Australasian Journal of Educational Technology, 2013, 29(6). 778 • time management (Clarke, 2004; Pallof & Pratt, 2003; Schrum & Hong, 2002); • reflection (Alexander & Boud, 2001; Pallof & Pratt, 2003); • computing and internet skills (Dupin-Bryant & DuCharme-Hansen, 2005; Salmon, 2002); • interaction skills (Hiltz, 1994; Mayes, 2001; Pirila & Yli-Luoma, 2007; Shea, Fredericksen, Pickett, Pelz, & Swan, 2001); • identity and social presence (Freeman & Bamford, 2004; McConnell, 2006; Tu, 2002; Tu & McIsaac, 2002). Effective e-learners are autonomous and take greater responsibility in the management of their own learning (Mayes & de Freitas, 2007; Stephenson, 2001). This includes decisions about when and where learning should take place; identifying learning outcomes (Stephenson, 2001); and determining how these learning outcomes will be achieved (Mayes & de Freitas, 2007). With increased emphasis on learner- created content, common in e-learning environments (Stephenson, 2001), students have to make decisions such as where to source content and how it should be distributed (TESEP, 2007). Students who lack these skills are likely to perform poorly in e-learning environments (Kearsley, 2000). Associated with student autonomy is self-direction, defined as the "skill of learning how to learn" (Dabbagh & Bannan-Ritland, 2005, p. 46). In constructivist e-learning environments, self-direction is perceived to be a critical skill as learners are afforded greater opportunity to take control of their own learning (Dabbagh & Bannan-Ritland, 2005). Students who are willing to take control perform more strongly in e-learning environments than students who do not (Alexander & Boud, 2001; Bauer et al., 2000; Hedberg, 2001; Kearsley, 2000). Self-awareness, self-monitoring, self-initiative, self-discipline and self-evaluation have all been identified as important strategies used by self-directed learners (Cheurprakobkit, Hale, & Olson, 2002). Time management is another strong predictor of student effectiveness (Loomis, 2000). Effective e- learners are characterised by their ability to set and prioritise goals (Clarke, 2004; Pallof & Pratt, 1999, 2003); plan study schedules (Loomis, 2000); balance work and family with study commitments (Clarke, 2004; Schrum & Hong, 2002); and avoid overload through the planning of regular breaks into schedules (Pallof & Pratt, 2003). Reflection can be defined as the ability of a learner to "plan and control one's learning processes" (Chung, Chung, & Severance, 1999, p. 238). In an e-learning environment, students have to be able to reflect and build a bridge between what they already know and what they have learned, thereby engaging and making sense of the content (Alexander & Boud, 2001). In online discussions, effective e-learners understand the importance of taking time to reflect upon the material they are learning or the ideas of their peers before composing a response (Pallof & Pratt, 2003). Not surprisingly, computing skill is an important factor determining effectiveness in e-learning environments (Dupin-Bryant & DuCharme-Hansen, 2005). According to Salmon (2002, p. 12), "there is a complex interplay between the participant's technical access and skills and the motivation to be active online." Student computing skills have been shown to vary greatly and to ensure that students are effective in online courses it has been recommended that computer skills development take place as early as possible (Dupin-Bryant & DuCharme-Hansen, 2005). Similar to computing skills, skills in using the Internet and associated applications have been identified as factors contributing to e-learner effectiveness. Dabbagh and Bannan-Ritland (2005) identified a variety of important web-based applications and the associated skills required to use them effectively. These included, being able to use browsers and search engines; understand URL configurations; locate websites; navigate through hyperlinks; evaluate web content; download and install plug-ins to view multimedia files; use tools that enable asynchronous and synchronous communication; and engage in collaborative and distributed learning activities. Interaction is a key element of e-learning environments built upon social constructivist principles (Bauer et al., 2000). Interaction can be defined as communication or dialogue that takes place between learners and instructors, learners and other learners, and users and the technology (Dabbagh & Bannan-Ritland, 2005). Interaction has been demonstrated to be an important determinant of effectiveness in e-learning environments (Hiltz, 1994; Mayes, 2001; Pirila & Yli-Luoma, 2007). Pirila and Yli-Luoma (2007) found Australasian Journal of Educational Technology, 2013, 29(6). 779 that learner-learner interaction had a significant relationship with student learning outcomes. Shea, Fredericksen, Pickett, Pelz, and Swan (2001) found that students who reported the highest levels of learning also reported the highest levels and quality of interaction with both instructors and other students. According to Wenger (1998, p. 145), issues of identity are "integral aspects of a social theory of learning". The formation of identity is believed to be as critical in e-learning environments as the construction of knowledge (Mayes, 2001). Identity is a major determinant of an individual's decision whether or not to interact with others (Wenger, 1998). Students need to be able to establish a sense of identity online in order to build supporting and trusting relationships with other learners (Hewson & Hughes, 2005). Closely associated with identity is social presence. Social presence can be defined as the degree of feeling, perception and reaction to another in an e-learning environment (Tu & McIsaac, 2002). Social presence has been identified as having positive effects on the degree of interaction and collaboration online (Gunawardena, 1995; McConnell, 2006; Tu, 2002; Tu & McIsaac, 2002). Social presence is closely related to identity and studies have shown that students who were more able to project their identities online generally had a stronger social presence (Freeman & Bamford, 2004; Tu, 2002). A common element in much of the e-learning literature on competencies is that much of it describes what students have to be (e.g., self-directed, self-aware) rather than what students need to do. There is a tendency in the literature to describe traits and characteristics rather than observable and measurable behaviours. Consequently, lists of assembled competencies often lack practical application. Furthermore, factors that determine student effectiveness are interrelated and do not function independently (Schrum & Hong, 2002). So rather than simply considering e-learners as a composite of isolated skills, a more complex description is required. Such a profile would be useful as it would provide a richer and more holistic picture of e-learners and the competencies they require to be effective. However, general profiles of e-learners presented as a composite list of competencies are rare (de la Teja & Spannaus, 2008). There is also little evidence in the literature to suggest that the relative importance of e-learning competencies have ever been assessed. Although the literature may identify given sets of competencies as being important, nothing appears to have been done to assess the relative importance of these competencies. Thus, describing a set of e-learning competencies in behaviour specific terms and assessing the relative importance of these competencies were considered activities worthy of pursuing. Research Questions The aim of the study was twofold: first, to develop a general profile of the e-learner by identifying the competencies required for effective performance in a university e-learning environment; and second, to determine whether the e-learning competencies identified were of equal importance. With these two aims in mind, the research questions were: • What competencies are considered essential for e-learning? • What is the relative importance of these e-learning competencies as perceived by e-learning stakeholders? Method Context of the study and participants The study focused on the competencies required for e-learning at the university-level. All participants in the study were drawn from a university in rural Australia. Its location away from large urban centres of the population meant that much of the teaching at this university was delivered externally and online. As it is common practice to define e-learning according to the context within which it is being implemented (Chadha & Kumail, 2002), for the purpose of the study e-learning was defined as learning mediated by a learning management system. This was considered an appropriate definition because learning management systems are the primary means by which e-learning is delivered in the university Australasian Journal of Educational Technology, 2013, 29(6). 780 sector both currently, and in the immediate future (Bonk, 2004; Coates, James, & Baldwin 2005; Siemens 2006). The online learning at the study site was delivered through a learning management system. As learning management systems do not create learning in and of themselves it was necessary to situate and develop the e-learning competencies within an appropriate learning paradigm. Social constructivism was selected as the paradigm due to its complementarity with e-learning. This is because social constructivism provides e-learning with a firm pedagogical foundation while e-learning environments can provide good opportunities for interaction and collaboration (Bauer et al., 2000). Social constructivism is based on the ideas of Vygotsky (1978) and the premise that learning originates and develops through social and cultural interaction (Gillani, 2003). Concepts developed at the social level become integrated at the individual level via a process referred to as "internalisation" (Vygotsky, 1978). Learning occurs through the interaction, discussion and the sharing of ideas, which supports the construction of new knowledge (Yuen & Chow, 2000). Study methodology The e-learning competencies were identified using a process known as the Hybrid Behaviourally Anchored Rating scale (Hybrid BARS). Hybrid BARS is a modified version of the Behaviourally Anchored Rating scale (BARS) first introduced by Smith and Kendall (1963) to assist in rating the performance of staff nurses. Hybrid BARS has the capacity to identify competencies reflecting effective performance in a particular area. Critically, Hybrid BARS is able to describe these competencies in "multidimensional, behaviour specific terms" (Anshel & Webb, 1991, p. 32). Hybrid BARS has been used in a variety of areas such as sports refereeing (Anshel, 1995; Anshel & Webb, 1991; Dickson, 2000), football coaching (Anshel, Housner, & Cyrs, 1987), primary school teaching (Moore & Webb, 1995) and classroom management and discipline (Jessup & Webb, 1994). No previous studies were found to have used the Hybrid BARS process to identify e-learning competencies. The Hybrid BARS process used in the study was implemented in five stages: Stage 1: Selection and formation of two expert panels Stage 2: Generation of e-learning competencies by these panels Stage 3: Amalgamation of lists by researcher Stage 4: Verification of amalgamated list by panel members Stage 5: External validation of the e-learning competencies These five stages were implemented as two distinct phases with each phase having a different objective and making use of different sets of participants. Phase one of the study involved the implementation of Stages 1 to 4 of the Hybrid BARS process. Central to the first phase was the meeting of two expert panels to identify the competencies considered essential for a university e-learning environment. These competencies formed the basis of the survey instrument implemented in Phase two of the study. Phase two involved the implementation of Stage 5 of the Hybrid BARS process. Using a survey instrument based upon the e-learning competencies identified in Phase one, respondents were asked to rate the importance of each of the e-learning competencies. Stage 1: Selection and formation of two expert panels The purpose of the expert panel is to develop competencies for the role under review (Cyrs, Dobbert, & Gussing, 1976). For the study an expert was defined as someone who had extensive knowledge, familiarity, expertise, or previous experience with e-learning. To ensure that the expert panels formed represented a good cross section of e-learning stakeholders, each panel was composed of the following: • former student experienced in e-learning; • lecturer in either higher education or adult education; • lecturer in Information Technology education; • educational developer; • two lecturers experienced in e-learning. Australasian Journal of Educational Technology, 2013, 29(6). 781 This made a total of six participants per panel. Expressions of interest were sent out to potential participants and two panels of experts were formed to take part in Stage 2 of the Hybrid BARS process. Stage 2: Generation of e-learning competencies by these panels Stage 2 involved the convening of two expert panel workshops during which the e-learning competencies were generated. The suggested format for the workshops was based upon a number of previous studies identifying competencies (see Anshel & Webb, 1991; Cyrs, 1979; Dickson, 2000). The two panel workshops were held three weeks apart to suit the work schedules of all participants. Each panel workshop was held over a five-hour period and involved working through a cycle of clarification, expansion and modification until consensus could be achieved between panel members of the competencies they considered would be indicative of effective performance in a university e-learning environment. To begin the process, panel members were asked to consider in broad terms the tasks students would be expected to undertake in a university e-learning environment. These broad level tasks are referred to as performance dimensions in the BARS literature (Smith & Kendall, 1963). Having defined a set of performance dimensions for e-learning, panel members were asked to develop competencies that they believed were indicative of effective performance within each of the defined performance dimensions. The outcome of each of the panel workshops was a draft list of e-learning competencies. Stage 3: Amalgamation of expert panel lists At the conclusion of the expert panel workshops, two distinctive lists of e-learning competencies (one from each panel) had been developed. Stage 3 involved the amalgamation of the two panel lists by the first author into a single list of e-learning competencies. Both lists were examined and where necessary, a number of changes as recommended by Cyrs et al. (1976) were made. These changes were: • duplicate phrases combined; • unclear statements clarified; • unnecessary statements deleted; • items added to complete deficient statements. Care was taken not to change the meaning or intent of the statements, but rather to provide a consistency of both language and format. The outcome of Stage 3 was a draft list of the amalgamated e-learning competencies. Stage 4: Verification of amalgamated list by panel members The purpose of Stage 4 was to ensure the integrity of the e-learning competencies by sending the amalgamated list of competencies to all panel members for review and approval. Panel members were asked to comment on this draft list with regard to the following questions (Dickson, 2000): • Are there any performance dimensions you feel need to be added, combined or deleted? • Do the competencies reflect accurately the performance dimension with which they are associated? • Are there any competencies you feel are superfluous? • Are there any competencies you feel are ambiguous? • Are there any competencies you feel would need to be reworded for clarification purposes? On the basis of feedback from panel members, a number of competencies were reworded and duplicate items removed. The final product was a list of the competencies panel members believed to be essential for effective performance in a university e-learning environment. This completed Phase one of the study. Australasian Journal of Educational Technology, 2013, 29(6). 782 Stage 5: External validation of the e-learning competencies Phase two of the study involved the implementation of Stage 5 of the Hybrid BARS process. Using an online survey instrument based upon the e-learning competencies identified in Phase one, the e-learning competencies were externally validated by 20 off-campus students and 15 staff members at the university. The student cohort was made up of postgraduate students who were either currently studying or had previously studied in fully online units. The staff cohort was made up of lecturers and educational developers. Phase two of the study made use of a different set of participants to Phase one. As part of the validation process, e-learning stakeholders were asked to rate the relative importance of the e-learning competencies using the following five point Likert scale items: 1 (irrelevant) – 2. (not very essential) – 3. (somewhat essential) – 4. (essential) – 5. (very essential). Classification of the e-learning competencies according to importance. The e-learning competencies were classified as Essential (must have), Important (should have), or Unimportant, using cumulative percentages calculated from the respondents' importance ratings. The terminology and criteria for classification into these categories were the same as used in previous Hybrid BARS studies (see Anshel, 1995; Anshel et al., 1987; Dickson, 2000; Moore, Webb, & Dickson, 1997). The criteria for this classification were as follows: • Essential (must have) At least 90% of responses ranked at either 4 (essential) or 5 (very essential). • Important (should have) At least 90% of responses ranked at 3 (somewhat essential), 4 (essential), or 5 (very essential). • Unimportant Failure to meet the above criteria. To facilitate analysis, the identified competencies were grouped into three categories based upon those used by Birch (2002). This was done as both a convenient means of organising the competencies and to help determine whether any patterns or trends could be identified in the importance ratings of the competencies. These categories are presented in Table 1 Table 1 Classification of e-learning competencies 1. management of learning and the e-learning environment; 2. interaction with the learning content; 3. interaction with the e-learning community. Results In Phase one of the study, expert panels members identified 58 e-learning competencies. Similar to previous implementations of the Hybrid BARS process (see, Anshel, 1995; Anshel & Webb, 1991; Dickson, 2000), the competencies identified by expert panel members were considered as being of equal importance; in other words all Essential. However, in Phase two of the study, cumulative percentages calculated from the importance ratings of 35 e-learning stakeholders surveyed indicated that the e- learning competencies were not of equal importance. For the 58 competencies identified, 19 were considered as Essential competencies (Table 2), 29 were considered as Important (Table 3), and 10 were considered Unimportant (Table 4). The e-learning competencies presented in Tables 2-4 have been grouped according to the categories identified by Birch (2002) and have been sorted alphabetically as to avoid implying a hierarchy to their order. Australasian Journal of Educational Technology, 2013, 29(6). 783 Table 2 Competencies rated as Essential (must have) 1. Management of learning and the e-learning environment balances work, social, family and study commitments downloads and uploads information and resources T identifies the requirements necessary to complete a task plans an appropriate strategy to undertake a task prioritises competing tasks within the time available searches the Internet strategically selects the appropriate technology tool for the task at hand T uses a web browser with skill and purpose T uses search engines effectively T 2. Interaction with the learning content able to distinguish between relevant and irrelevant items able to navigate large bodies of content T develops responses which synthesise a range of ideas forms connections between prior knowledge and new knowledge identifies and rectifies gaps in one's own understanding reads and writes at an appropriate level 3. Interaction with the e-learning community asks for guidance or seeks clarification for misunderstandings provides responses in clear, concise and unambiguous language responds to others with respect seeks information through either own enquiries or the questioning of others Note. T = related to the use of technology Of the 58 e-learning competencies identified, 22 were related to the use of technology (indicated by a superscripted T in Tables 2-4). The remaining 36 competencies encompassed practices that could be considered as being essential to learning situated within the social constructivist paradigm. The e-learning competencies identified were generally consistent with those previously identified in the literature. However, six competencies were either new or significantly different from what had been previously identified. These were: • acknowledges the facilitation role of the lecturer in the learning environment; • critiques a web site in relation to content; • critiques the responses of others constructively; • evaluates a set of search results critically; • makes allowances for the virtual nature of the learning environment; • recognises lecturer's response as a contribution and not the final word on an issue. The competencies considered as Unimportant represented 17% of the total number of competencies identified. A previous Hybrid BARS study by Dickson (2000), reported a value of 11% for competencies rated as "Unimportant". Dickson argued that this could be interpreted as an indication of broad acceptance by the stakeholders of the competencies identified by the expert panels. The figure of 17% was of similar magnitude to the value reported by Dickson, and as such, this result can also be taken as an indication of a broad acceptance of the e-learning competencies by the e-learning stakeholders. Table 5 shows the breakdown of importance ratings according to the three broad categories of e-learning competencies. Table 5 reveals a number of significant points with respect to the importance ratings. First, the high level of importance given to competencies in the "Management of learning and the e-learning environment" and "Interaction with the learning content categories" made up 79% of the Essential (must have) competencies and 69% of the Important (should have) competencies. Second, eight of the ten competencies (80%) rated by stakeholders as Unimportant were grouped in the "Interaction with the e- learning community" category. Australasian Journal of Educational Technology, 2013, 29(6). 784 Table 3 Competencies rated as Important (should have) 1. Management of learning and the e-learning environment adapts learning style to the e-learning environment T anticipates and makes allowances for "wait time" in asynchronous discussions T demonstrates knowledge and use of the learning management system T employs a logical process to identify and solve a computer problem T engages in the process of reflection makes allowances for the virtual nature of the learning environment T understands own cognitive processes and thinking strategies undertakes set tasks independently uses feedback to evaluate own performance (self-critique) uses problem solving strategies uses technology to assist in the construction of knowledge T uses technology to support own learning style T views oneself positively as a learner works to a disciplined timeframe 2. Interaction with the learning content accesses information from a variety of sources (e.g., web pages, podcasts) T critiques a web site in relation to content T cross-references between sources to determine accuracy evaluates a set of search results critically T extracts information from a variety of formats T goes outside the technology and learning community to seek information T 3. Interaction with the e-learning community acknowledges the facilitation role of the lecturer in the learning environment T applies the rules of netiquette consistently T considers and acts upon feedback from members of the learning community critiques the responses of others constructively determines when it's time to "listen" to or contribute a response justifies own stance on an issue recognises lecturer's response as a contribution and not the final word on an issue uses inter-personal communication skills willing to have ideas challenged Note. T = related to the use of technology Table 4 Competencies rated as Unimportant 1. Management of learning and the e-learning environment integrates a variety of software applications to create a product T 2. Interaction with the learning content presents information in a variety of formats (video, audio, etc.) T 3. Interaction with the e-learning community arranges schedule to allow for regular online sessions T comments upon or critiques a response made by the lecturer contributes new ideas to a discussion encourages others to post through positive responses T seeks interaction with other members of the learning community shares personal experiences when relating to topic and others views oneself as a member of the learning community works with others to collaboratively construct knowledge Note. T = related to the use of technology Australasian Journal of Educational Technology, 2013, 29(6). 785 Table 5 Breakdown of e-learning competencies according to importance ratings Management of learning and the e- learning environment Interaction with the learning content Interaction with the e-learning community Essential (must have) (n=19) 9 (47%) 6 (32%) 4 (21%) Important (should have) (n=29) 14 (48%) 6 (21%) 9 (31%) Unimportant (n=10) 1 (10%) 1 (10%) 8 (80%) Discussion Phase one: Identification of the e-learning competencies Less than half (38%) of the e-learning competencies identified by the expert panels in Phase one of the study were specifically related to the use of technology. The remaining competencies were associated with practices that could be considered as important within the social constructivist paradigm. This alignment of the e-learning competencies with social constructivism raises the issue of whether the identified competencies represent the full suite of competencies for e-learning. It could be argued that as the request was made during the expert panel workshops for panel members to develop e-learning competencies within a social constructivist framework, the competencies identified arose as the simple consequence of this request. However, due to the nature of the Hybrid BARS process, any competency that emerged from the expert panel deliberations and was subsequently endorsed by panel members would have been included irrespective of whether or not it was aligned with social constructivism. The six competencies identified that were either new or significantly different from what had been identified previously in the literature perhaps reflect the changing nature of teaching in e-learning environments. In particular, the shift from teacher (lecturer) centred to student centred learning. In practical terms, this represents a change in the instructional role of lecturers towards one characterised by facilitation. Three of the six competencies – "critiques the responses of others constructively", "evaluates a set of search results critically" and "critiques a web site in relation to content" – deal with critique and evaluation. As students spend more time sourcing their own content from the Internet rather than from more traditional sources, evidenced by the increasing number of citations coming from sites such as Wikipedia, students need the ability to be critical about the information they access and use. Two competencies – "acknowledges the facilitation role of the lecturer in the learning environment" and "recognises lecturer's response as a contribution and not the final word on an issue" – reflect the changing role of lecturers in e-learning environments, possibly heralding the end of the sage-on-the-stage role of lecturers to one that is more the guide-on-the-side. The final competency of the six – "makes allowances for the virtual nature of the learning environment" – reflects the differences between real-time and virtual learning environments, acknowledging that there are differences between them and that these need to be accommodated. Phase two: Importance ratings of the e-learning competencies Despite a reasonable level of support by the e-learning stakeholders for the e-learning competencies in general (Table 5), importance ratings were not evenly distributed across the three categories of e-learning competencies. Almost half (47%) of the competencies rated as Essential were from the "Management of learning and the e-learning environment" category. This category also accounted for almost half (48%) of the competencies rated as Important. Many of the competencies in this category were associated with skills in organisation and planning. For example, "balances work, social, family and study commitments", "identifies the requirements necessary to complete a task", "plans an appropriate strategy to undertake a task" and "prioritises competing tasks within the time available". This suggests that giving students information about how to organise and plan their learning can be an important determinant of their success online. This information could be in the form of generalised advice before students begin a course Australasian Journal of Educational Technology, 2013, 29(6). 786 of study or advice related to specific aspects of their studies. Certainly, in an e-learning environment, the focus should not simply be on what has to be learned but also on how it might best be learned. Importantly, eight of the ten competencies (80%) rated as being Unimportant were grouped in the "Interaction with the e-learning community" category. While there was broad acceptance of the e-learning competencies in general, many of the competencies dealing with working and interacting with others did not receive such widespread support from e-learning stakeholders. Closer inspection of the importance ratings of the competencies in the "Interaction with the e-learning community" category shows that those competencies that did rate highly in importance were generally related to how e-learners responded to others as opposed to how e-learners worked with others, which were rated as being less important. For example, the competencies "responds to others with respect" and "provides responses in clear, concise and unambiguous language" were rated as Essential competencies, while the competencies "contributes new ideas to a discussion" and "works with others to collaboratively construct knowledge" were rated as Unimportant. This is despite these competencies being identified as essential by the expert panels in Phase one. Critically, the literature also has identified competencies such as these as important as they are considered central to the ideas of social constructivism. The lack of support for competencies involving interaction with members of the e-learning community and the importance placed upon activities focused on managing the learning and interacting with content suggests that there is a disconnect between what the literature says about the importance of social constructivism to e-learning environments in theory and what the e-learning stakeholders perceive its importance to be in practice. This is not unusual, because although constructivism is the dominant espoused theory in higher education it is commonly not the dominant theory-in-use for computer-based learning environments (Jackson, 1998). The results from this study suggest that this is also the case for e- learning environments mediated by learning management systems. The critical question is what has caused such a disconnect? One possible hypothesis is that few university lecturers have ever been e-learners (Barnes & Tynan, 2007) and this could make it challenging for them to design and teach in learning environments in which they have not had firsthand experience as learners. As a result, it may be difficult for lecturers to envisage what learning in an e-learning environment developed in accordance with social constructivist principles is actually like. Also, as suggested by the importance ratings, students appear not to be fully apprised with this type of learning either. If they were, then it might be expected that the competencies involved with working and interacting with others would be rated of higher importance. However, if experiences of these type of e-learning environments are lacking or have not been positive, then it is not surprising that many of the e-learning competencies considered pivotal from a social constructivist perspective were considered by the e-learning stakeholders as being unimportant. Hence, the disconnect between what the literature says in theory and what e- learning stakeholders believe in practice. One means by which this disconnect between theory and practice might be bridged is by having both students and staff being apprenticed into a learning culture of social constructivism. For students, one of the precursors for effective learning within a university e-learning environment would be helping them to learn how learning best occurs in these environments. At the study site for instance, programs have been put in place to assist students in developing a range of academic skills such as subject specific writing, academic reading, academic writing, learning strategies, and time management (Academic Skills Office, n.d.). However, there appears to be little support available to assist students in learning how to learn in an e-learning environment. The results of this study suggest that the development of such programs is warranted. Correspondingly, staff need to be apprenticed into a teaching culture of social constructivism. It is sometimes easy to forget that staff are users of e-learning systems as well and like students, also bring with them assumptions, motives, intentions, and previous knowledge that determine the quality of the teaching they deliver. Learning design provided or mediated by specialist learning designers is one possible means by which staff might be apprenticed into a teaching culture of social constructivism. According to Campbell, Schwier, and Kenny (2009), learning design has the potential to transform institutional teaching practice. The very kind of learning that universities aim to provide students – authentic, real life, cognitively demanding and embedded in social experience – in other words, in accordance with social constructivist principles, exists when learning designers work collaboratively with Australasian Journal of Educational Technology, 2013, 29(6). 787 their clients (Campbell et al., 2009). In the university context, with learning designers working with lecturers, effective e-learning environments could be created and through this process, lecturers could experience learning built upon social constructivist principles. This learning could be passed on to students. Such working relationships need to be encouraged and developed because simply dropping content into a learning management system will not ensure good learning. It is not the container that is critical in ensuring high quality e-learning environments but rather, good learning design developed in accordance with learning theory. Implications and future research The implications of the results of this study for e-learning are largely dependent upon the degree to which the findings of the study can be extrapolated beyond that of the study site. The e-learning competencies were developed specifically for learning mediated by a learning management system in a university e- learning environment in accordance with social constructivist principles. This level of specificity was a requirement of the Hybrid BARS process to ensure that the competencies identified were an accurate and comprehensive reflection of all aspects of effective performance. However, this process has the potential to limit the study by restricting the applicability of the e-learning competencies to the site where they were developed. To ensure applicability of the e-learning competencies beyond the study site, the context of the study – a university learning management system – was chosen deliberately, not simply because one was in use at the study site, but because learning management systems remain the most common means of delivery of e-learning in universities (Coates et al., 2005; Siemens, 2006). Importantly, the use of learning management systems in universities is expected to increase in the future (Bonk, 2004). The choice of learning theory in which to situate the e-learning competencies – social constructivism – was also deliberate, as this theory is widely accepted as being the most appropriate learning theory for informing e- learning practice (Bauer et al., 2000). Consequently, the e-learning competencies identified by this study should be of relevance to the broader e-learning community to help inform learning design or be used as the basis for training and development programs. It must be acknowledged that the scope of the study was restricted to a single institution and this could potentially limit the study's findings. Other factors, such as the culture of teaching and learning unique to the institution; level of study - undergraduate or postgraduate; type of assessment undertaken; computing expertise and previous e-learning experience, may have influenced the view of e-learning held by the study participants. Determining the impact of such factors would be avenues for future research. Research focusing on a broader assessment of the e-learning competencies across other institutions is also necessary to help determine the generalizability of the identified e-learning competencies. It would also be worthwhile to repeat the external validation survey at the institution where the e-learning competencies were developed. As one of the findings of the study was that stakeholders had a view of e-learning that was not fully consistent with the principles of social constructivism, it would be interesting to determine the degree to which this view persists amongst students and staff at the study site. Conclusion This paper reported the results of a study designed to identify the competencies required for effective performance in a university e-learning environment. Using a procedure known as Hybrid BARS, 58 competencies were identified which were considered to be essential for effective performance in a university e-learning environment set within the context of a learning management system and situated within the social constructivist learning paradigm. The e-learning competencies identified were generally consistent with those previously identified in the literature. However, six competencies were either new or significantly different from what had been previously identified. The e-learning competencies identified in the current study describe in observable and measurable terms the requisite knowledge, understandings, skills, attitudes and behaviours students required for effective performance. By describing e-learning competencies in this manner, the tendency in the e-learning competency literature to focus on traits and characteristics was able to be avoided. Australasian Journal of Educational Technology, 2013, 29(6). 788 Importance ratings by e-learning stakeholders categorised the identified e-learning competences in terms of their perceived level of importance. Despite social constructivism being considered the predominant learning theory informing e-learning, many of the e-learning competencies associated with interacting and working with others were considered unimportant by e-learning stakeholders. This suggests that there is a disconnect between how the importance of social constructivism is considered in theory and how e- learning stakeholders perceive its importance to be in practice. References Academic Skills Office. (n.d.). Academic skills online: tUNEup. Retrieved from http://www.une.edu.au/tlc/aso/students/programs/tuneup/ Alexander, S., & Boud, D. (2001). Learners still learn from experience when online. In J. Stephenson (Ed.), Teaching and learning online: Pedagogies for new technologies (pp. 3-15). London: Kogan Page. Anshel, M. (1995). Development of a rating scale for determining competence in basketball referees: Implications for sport psychology. The Sports Psychologist, 9(1), 4-28. Anshel, M., Housner, L. D., & Cyrs, T. E. (1987). Defining competence for effective coaching of high school football. Journal of Applied Research in Coaching and Athletics, 2(2), 79-95. Anshel, M., & Webb, P. (1991). Defining competencies for effective refereeing. Sports Coach, 15(3), 33- 37. Barnes, C., & Tynan, B. (2007). The adventures of Miranda in the brave new world: Learning in a Web 2.0 millennium. ALT-J Research in Learning Technology, 15(3), 189-200. Bauer, K., Chin, K., & Chang, V. (2000, July). Web-based learning: Aspects of cultural differences. Paper presented at the European Conference on Information Systems, Vienna, Austria. Retrieved from http://is2.lse.ac.uk/asp/aspecis/20000068.pdf Birch, P. (2002). Elearner competencies. Retrieved from http://sdl_demo.s3.amazonaws.com/89/www/24328/1/1.pdf Bonk, C. (2004). The perfect E-Storm emerging technology, enormous learner demand, enhanced pedagogy, and erased budgets Part 1: Storms #1 and #2. The Observatory on Borderless Higher Education. Retrieved from http://www.obhe.ac.uk/products/reports/publicaccesspdf/Bonk.pdf Campbell, K., Schwier, R., & Kenny, R. (2009). The critical, relational practice of instructional design in higher education: an emerging model of change agency. Educational Technology Research and Development, 57(5), 645-66. Chadha, G., & Kumail, S. M. N. (2002). Elearning: An expression of the knowledge economy. New Delhi: McGraw-Hill. Cheurprakobkit, S., Hale, D., & Olson, J. (2002). Technicians' perceptions about web-based courses: The University of Texas system experience. American Journal of Distance Education, 16(4), 245-258. Chung, S., Chung, M., & Severance, C. (1999). Design of support tools and knowledge building in a virtual university course: Effect of reflection and self-explanation prompts. In P. De Bra & J. Leggett (Eds.), Proceedings of WebNet World Conference on the WWW and Internet 1999 (pp. 237-242). Chesapeake, VA: AACE. Clarke, A. (2004). E-learning skills. Basingstoke:UK: Palgrave. Australasian Journal of Educational Technology, 2013, 29(6). 789 Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11, 19-36. Cocanougher, A. B., & Ivancevich, J. M. (1978). 'Bars' performance rating for sales force personnel. The Journal of Marketing, 42(3), 87-95. Cyrs, T. (1979). Derivation of competence needs for training directors through consensual judgements workshop. Chicago: The University of Chicago Human Resource Center. Cyrs, T., Dobbert, D., & Grussing, P. (1976). Progress report: Competency identification including evaluation and validation. Minneapolis: College of Pharmacy, University of Minnesota. Dabbagh, N., & Bannan-Ritland, B. (2005). Online learning: Concepts, strategies and application. Upper Saddle River, NJ: Pearson. de la Teja, I., & Spannaus, T. (2008). New online technologies: New online learner competencies. Really? In J. Visser & M. Visser-Vafrey (Eds.), Learners in a changing learning landscape (pp. 187- 211). Netherlands: Springer. Dickson, S. H. (2000). A multifaceted analysis of performance criteria: Implications for the education and assessment of elite soccer officials in Australia (Unpublished doctoral dissertation). University of New England, Australia. Dupin-Bryant, P., & DuCharme-Hansen, B. (2005). Assessing student needs in web based distance education [Electronic Version]. Journal of Instructional Technology and Distance Learning, 2(1). Retrieved from http://itdl.org/journal/Jan_05/article04.htm Freeman, M., & Bamford, A. (2004). Student choice of anonymity for learner identity in online learning discussion forums. International Journal on E-learning, 3(3), 45-53. Gillani, B. (2003). Learning theories and the design of e-learning environments. Lanham, Maryland: University Press of America. Gunawardena, C. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1(2/3), 147-166. Hedberg, J. (2001). The online and digital experience: Reassuring higher education outcomes. In L. Vandervert, L. Shavinina, & R. Cornell (Eds.), Cybereducation: The future of long-distance learning. Larchmont, NY: M. A. Liebert. Hewson, L., & Hughes, C. (2005). Social processes and pedagogy in online learning. AACE Journal, 13(2), 99-125. Retrieved from http://www.editlib.org/index.cfm/files/paper_4805.pdf Hiltz, S. (1994). The virtual classroom: Learning without limits via computer networks. Norwood, NJ: Ablex. Hoffmann, T. (1999). The meanings of competency. Journal of European Industrial Training, 23(6), 275- 285. Jackson, B. (1998). Evaluation of learning technology implementation. Retrieved from http://www.icbl.hw.ac.uk/ltdi/evalstudies/esevalimp.pdf Jessup, S., & Webb, P. (1994). Teacher perceptions of the essential skills of classroom management and discipline. Paper presented at the Australian Association for Research in Education, Newcastle, Australia. Retrieved from http://publications.aare.edu.au/94pap/jesss94013.txt Australasian Journal of Educational Technology, 2013, 29(6). 790 Kearsley, G. (2000). Online education: Learning and teaching in cyberspace. Belmont, CA: Wadsworth. Loomis, K. (2000). Learning styles and asynchronous learning: Comparing the LASSI model to class performance. Journal of Asynchronous Learning Networks, 4(1), 22-32. Mayes, T. (2001). Learning technology and learning relationships. In J. Stephenson (Ed.), Teaching and learning online: Pedagogies for new technologies (pp. 16-26). London: Kogan Page. Mayes, T., & de Freitas, S. (2007). Learning and e-learning. In H. Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital age: Designing and delivering e-learning (pp. 13-25). Oxon, UK: Routledge. McConnell, D. (2006). E-learning Groups and Communities. Maidenhead: England: Open University Press. Moore, D., & Webb, P. (1995, November). Roles and competencies required by primary teachers in teaching personal development, health and physical education. Paper presented at the Australian Association for Research in Education, Hobart Australia. Retrieved from http://publications.aare.edu.au/95pap/webbp95036.txt Moore, D., Webb, P., & Dickson, S. (1997). Perceptions of preservice primary teachers in teaching personal development, health and physical education In J. Walkulski, S. Wright, & S. Tan Kwang San (Eds.), AIISEP World Conference on Teaching, Coaching and Fitness Needs in Physical Education and the Sports Sciences (pp. 144-152). Nanyang Technological University, Singapore. Pallof, R., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass. Pallof, R., & Pratt, K. (2003). The virtual student: A profile guide to working with online learners. San Francisco: Jossey-Bass. Parry, S. (1998). Just what is a competency (and why should you care?). Training, 35(6), 58-64. Pirila, K., & Yli-Luoma, P. (2007). Learning outcomes, interaction and motivation in e-learning. In G. Richards (Ed.), Proceedings of World Conference on E-learning in Corporate, Government, Healthcare, and Higher Education 2007 (pp. 6460-6467). Chesapeake, VA: AACE. Salmon, G. (2002). E-tivities: The key to active online learning. London: Kogan-Page. Schrum, L., & Hong, S. (2002). Dimensions and strategies for online success: Voices from experienced educators. Journal of Asynchronous Learning Networks, 6(1), 57-67. Shea, P., Fredericksen, E., Pickett, A., Pelz, W., & Swan, K. (2001). Measures of learning effectiveness in the SUNY Learning Network. Retrieved from http://tlt.suny.edu/originaldocumentation/library/researchsln/ALNWorkshop2000.pdf Siemens, G. (2006). Learning or management system? A review of learning management system reviews. Retrieved from http://ltc.umanitoba.ca/wordpress/wp-content/uploads/2006/10/learning-or- management-system-with-reference-list.doc Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for ratings scales. Journal of Applied Psychology, 47(2), 149-155. Stephenson, J. (2001). Learner managed learning - an emerging pedagogy for learning online. In J. Stephenson (Ed.), Teaching and learning online: Pedagogies for new technologies. London: Kogan Page. Australasian Journal of Educational Technology, 2013, 29(6). 791 Transforming and Enhancing the Student Experience through Pedagogy (TESEP) (2007). Re-thinking learning technology. Retrieved from http://www2.napier.ac.uk/transform/Rethinking_Learning_Technology.pdf Tu, C. (2002). The measurement of social presence in an online learning environment. International Journal on E-learning, 1(2), 34-45. Tu, C., & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. The American Journal of Distance Education, 16(3), 131-150. Vygotsky, L. (1978). Mind in society (M. Cole, Trans.). Cambridge, MA: Harvard University Press. Wenger, E. (1998). Communities of practice: Learning, meaning and identity. Cambridge, UK: Cambridge University Press. Yuen, H., & Chow, Y. (2000). Social-Constructivist approach. In N. Law, H. K. Yuen, W. W. Ki, S. C. Li, Y. Lee & Y. Chow (Eds.), Changing Classroom and Changing Schools: Study of Good Practices in Using ICT in Hong Kong Schools (pp. 103 - 113). Hong Kong Centre for Information Technology in School and Teacher Education, The University of Hong Kong. Corresponding author: Mitchell Parkes, mparkes2@une.edu.au Australasian Journal of Educational Technology © 2013. Please cite as: Parkes, M., Reading, C., & Stein, S. (2013). The competencies required for effective performance in a university e-learning environment. Australasian Journal of Educational Technology, 29(6), 777-791.