piguet Australian Journal of Educational Technology 2000, 16(3), 302-314 Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability Allison Piguet and Daniel Peraya TECFA, University of Geneva This study focuses on the creation of web-integrated learning environments, using WebCT [http://www.webct.com] More specifically, it targets the user, as a course designer, who interacts with several authoring tools in order to produce educational scenarios. Two factors were measured in terms of their significance within the user-interface interaction: user satisfaction and control. A positive linear dependence between users' perception of satisfaction and control was observed for a variety of WebCT tools. In conclusion, this positive correlation could be explained both quantitatively, and qualitatively in terms of a tool's perceived aspect of usability. Introduction The advances in computer development along with the expansion of the world wide web have lead to the creation of new and innovative designs in technology based instruction. Today there are numerous authoring tools that have broadened the base of possible instructional developers and allowed non-programmers, especially teachers, to create their own instructional learning environments. Authoring tools, also referred to as course design tools, provide a vital purpose that give course designers the necessary utilities to create unique integrated web-integrated learning environments, without knowing extensive programming languages. This study highlights course designers using WebCT and their interaction with the warranted authoring tools, fundamental for creating web- integrated learning environments. Therefore, a closer look is given to the utilisation of various WebCT tools that positively influence these course designers by evoking high perceptions of control and overall satisfaction. More specifically, this study combines these perceptions of user control vs. Piguet and Peraya 303 overall satisfaction and applies them to further analyse the aspect of usability according to specific WebCT authoring tools. These two variables, control and satisfaction were measured both quantitatively and qualitatively, providing the basis for the experimentation, analysis, and discussion included in this research study. Additionally, the attribution of usability is explained by considering these two elements in the user- interface interaction. Fundamentals of a web-integrated learning environment Integrating all of the well established advantages of the world wide web (WWW), Wilson defines a learning environment as a place where learners may work together and support each other as they use a variety of tools and information resources in their pursuit of learning goals and problem solving activities (1995). A web based learning environment is a hypermedia based program or system that uses the attributes and resources of the WWW to facilitate learning. Likewise, it is a networked computer application that enables people to learn from a distance. Learners can be physically separated from teachers and from each other, and they can participate in the learning environment at their own time and pace. Simply, a web based learning environment is a place where learners and teachers interact. For the past two years, we have explored different authoring tools, both commercial and non-commercial that facilitate the creation of web- integrated learning environments. WebCT, was a particular tool that attracted our interest for two reasons. First, was its ability to combine single function Internet based tools to create varies multifunctional tools that could suit our particular teaching and learning purposes at TECFA. Secondly, WebCT provides an authoring environment that portrays a "desktop model," where tools are available and unified from within a common restricted web based interface. As a result, we have implemented WebCT in one of the teaching and learning modules at TECFA. In the course, the students take the seat as the course designers and have the opportunity to author their own unique learning environments. Therefore, studying these two aspects, interaction between course designer and WebCT authoring tools, was of great interest to us in our research domain. Measuring usability in the human-computer interaction Usability is a reoccurring concept in the domain of human-computer interaction (HCI). Its popularity has given rise to numerous definitions concerning its functional meaning. One of the first to recognise the relativity of this concept was Shackel (1991). He defined usability as the 304 Australian Journal of Educational Technology, 2000, 16(3) capability of a system to be used by humans easily and effectively. Nielsen (1993), extended this definition to include that usability can be operationally defined as the effectiveness, efficiency and satisfaction with which specified users can perform particular tasks in a given environment. Accordingly, Preece (1994) mentions that usability is a measure of the ease with which a system can be learned or used, its safety, effectiveness and efficiency, and the attitude of its users towards it. Applying these definitions, there has been a continuing effort to further define all of the multiple dimensions of usability. More concretely, a set of usability principles has been explained and agreed upon by several HCI researchers, including Shneiderman (1986), Norman (1988), Nielsen (1993), and Shackel (1991). Each of their approaches defines the concept of usability by naming examples of system properties or qualities that influence usability (ie. consistency, presentation, error handling and recovery, memory load reduction, task match, flexibility, ...). These functional operations of usability are measured only in terms of the system’s operation. Besides from the usability of the system’s operation, it also equally important to consider the point of view of the user in regards to usability. Therefore an attempt to study the experience of a user by introducing approaches that link usability and human emotion can also be introduced. These include, both the attitudes and perceptions of the user in terms of satisfaction and control. Further, user satisfaction is explained as the affective attitude towards a particular computer application by a user who interacts with the application directly. Along with this, there is also a practical assumption that user satisfaction with an information system results in some positive change in user behaviour, resulting in increased effectiveness. Additionally, user participation in design was found to be positively correlated with user satisfaction (Doll and Torkzadeh 1988). On the other hand, user control in a workspace concerns those aspects of interface and instructional design that provide users with functions in order to choose the nature of the feedback they receive the navigation paths through the content, the content to be engaged in and the pace of the engagement (Reeves, 1993). It has been suggested that giving the user control over system, process, and content generally increases intrinsic motivation to learn (Becker & Dwyer 1994). Piguet and Peraya 305 Perception of usability In recent years, researchers have conducted several studies to examine the relationship between perceived ease of use, perceived usefulness, and the usage of other information technologies (Davis, 1989, Chau, 1996). Their research has supported the notion that perceived ease of use and perceived usefulness can predict the usage of technology. Fishbein and Ajzen’s (1975) theory of reasoned action shows that beliefs influence attitudes which lead to intentions and therefore generates behaviours. Davis (1989) explains this theory of Technology Acceptance Model, or TAM, in relation to both a user’s attitude toward use and his/her behavioural intention to use. Attitude towards use is the user’s evaluation of the desirability of employing a particular information systems application. Behavioural intention to use is a measure of the likelihood a user will employ the application (Ajzen and Fishbein, 1980). An important topic of concern for these theories is the issue of "perception," experienced by users when using authoring tools in a system. This study will focus on the users' "perception" of control and "perception" of satisfaction when using different tools. It will be considered that course designers construe and create their own unique way of interacting with tools and construct their own interpretation of usability based on their own perception of satisfaction and control. The present study Pursuing the component of user-interface interaction, it is particularly interesting to apply these concepts of perception of satisfaction and control in the creation of web-integrated learning environments. More specifically, a course designer’s interaction with the tools within the web-integrated environment WebCT, with respect to their usability. For this study, six specific course design tools were chosen and tested in the experiment. These tools reflected important pedagogical components for effective course design and delivery for web-based online courses conducted at a distance. These six selected tools are detailed in Table 1. Continuing, the purpose of this study was to investigate the following questions regarding the use of WebCT authoring tools: 306 Australian Journal of Educational Technology, 2000, 16(3) 1. What is the variation of a course designer’s perception of control and satisfaction between different types of WebCT authoring tools? 2. Do some WebCT authoring tools have a higher degree of usability than other types of tools? 3. What are the implications of these results for course designers using WebCT? Table 1: Six authoring tools were chosen from WebCT. These tools represented important pedagogical categories in the design and delivery of web based teaching and learning scenarios. Tool title Pedagogical classification Tool 1 Customise homepage Presentation tool Tool 2 Icons Presentation tool Tool 3 Course Settings Course Management tool Tool 4 Quizzes Student Evaluation/ tracking tool Tool 5 Bulletin Board Communication tool Tool 6 Course Calendar Course Management tool Method Subjects In order to find participants for the experiment, it was imperative to target users who currently use, or had used WebCT to design and create web- integrated learning environments. Therefore, lists of higher educational institutes, using WebCT in Europe, United States, and Australia were gathered, along with the names of the individuals responsible for overall course design. The targeted population of the experiment included a sample of individuals, representing higher educational institutes, who had developed web-integrated learning environments using WebCT. More specifically, the sample profile included 35 participants who took part in the experiment; 13 professors, 3 teaching assistants, 7 graduate students, and 12 computer administrators. 46% of the participants had previous experience using other environments similar to WebCT, and 54% had never worked with similar environments. The participants differed widely in their level of computer experience using WebCT; 14% had only been working on WebCT for less than 1 month, 15% had worked with WebCT Piguet and Peraya 307 for one to six months, 31% worked with WebCT for six months to one year, and 40% had worked with WebCT for one year or more. Additionally, the participants spent varied amounts of time each week working on WebCT: 17% spent less than one hour on WebCT, 23% spent one to four hours, 26% spent four to ten hours, and 34% spent over ten hours. Materials A web based questionnaire was developed by the author to analyse the 6 unique WebCT tools. Questions of overall usability were derived from the Computer System Usability Questionnaire (CSUQ), assessing both user satisfaction and usability (Lewis, 1995). Further, questions of control were adapted from Garrison and Baynton's (1987) concept of control that explains the extent to which learners experience independence, competence and support in distance education. In total, a seven point, seventy two item Likert scale was implied in the questionnaire. The questionnaire contained 6 different sections, each section corresponding to one of the chosen WebCT tools. Within each section, a series of check box type format and text areas for user comments were provided. The Likert type scale consisted of a series of declarative statements, asking the participant to indicate whether he/she agreed or disagreed with each statement. These statements represented a set of user attitude statements that reflected both overall usability and satisfaction plus a component of user perception of control. Some of the questions included: a) Whenever I make a mistake using the tool, I recover easily and quickly. b) It was easy to learn to use this tool. c) I am able to complete my work quickly using this tool. Each question item was rated on a scale from 1 to 7, ranging from 1 (strongly agree) to 7 (strongly disagree). In addition, "not applicable" was listed as an option. At the end of each set of questions for each tool, additional space was given that prompted the user to make qualitative comments regarding their personal experience using each tool. Procedure In total, 150 emails were sent to the targeted WebCT course designers, asking them to participate in a post-task questionnaire, concerning the six WebCT tools. Those who were willing to participate in the experiment were asked to fill out the web based questionnaire and to answer all of the questions that were posed, with the option of anonymity. Furthermore, the subjects were asked to reflect back on using six different WebCT tools and 308 Australian Journal of Educational Technology, 2000, 16(3) to answer each of the following questions by expressing agreement or disagreement on the seven point scale. When the questionnaire was completed, two button prompts were provided, giving the subject two options either to submit the results or to erase the results and start over. Each submitted result was collected in a database, facilitating the data analysis for the experiment. Results Reliability The calculation of the results for the quantitative data was based from psychometric theory, where scale reliability is a function of the inter- relatedness of scale items, the number of scales, the number of scale steps per item, and the number of items in a scale. If a participant chose not to answer an item, the effect slightly reduced the reliability of the scale. However, the remaining items offered a reasonable estimate of the appropriate scale score. From a practical standpoint, by averaging the answered items to obtain the scale score enhanced the flexibility of use of the questionnaire. This is due to the fact that if an item was not appropriate in a specific context and users chose not to answer it, the questionnaire still remained useful. Also, users who did not answer every item remained in the sample. Finally, averaging items to obtain scale scores did not affect the statistical properties of the scores, and standardised the range of scale scores (Nunnally, 1978). Additionally in this study, the questions were based on standardised 7- point scale containing a mid-point. This choice of using a 7-point scale is justified by Matell and Jacoby’s (1972) study that demonstrated that as the number of scale steps increases, the respondents’ use of the mid-point category decreases as well. However, studies have shown that scales without a mid-point would be preferable due to the phenomenon that is pushes more respondents towards the positive end of the scale (Worcester and Burns, 1975). Quantitative analysis The first step of the analysis was to distribute the means of the participant responses for the two factors of satisfaction and control for each of the tools. This procedure was accomplished by creating scatter plot graphs for the distributed means along the x-axis and y-axis. More precisely, satisfaction means were plotted along the x-axis and the control means Piguet and Peraya 309 were plotted along the y-axis. The resulting effects were six scatter plot graphs, all resulting in a positive linear correlation. (Table 2.) Table 2: The values for each of the scatter plot graphs for the six tools demonstrated that there was a positive and dependent relationship between satisfaction and control. More interestingly, for each of the tools, the value of slope b illustrated that a change in control directly influenced the change in satisfaction. Tool 1 positive linear correlation R = 0.799 slope b = 0.672 N=35 Tool 2 positive linear correlation R = 0.800 slope b = 0.761 N=32 Tool 3 positive linear correlation R = 0.830 slope b = 0.642 N=34 Tool 4 positive linear correlation R = 0.720 slope b = 0.574 N=30 Tool 5 positive linear correlation R = 0.821 slope b = 0.804 N=31 Tool 6 positive linear correlation R = 0.906 slope b = 0.785 N=29 Validity The results from the six scatter plot graphs and linear fits proved that there was a positive and dependent relationship between satisfaction and control for each of the six WebCT tools. However, the results did not statistically quantify the differences between the slope values for each of the different tools. Therefore, a two population, independent, t-test was conducted (given a significance alpha level of 0.05) to test these slope differences. The results from the t-test proposed that the slopes between the different tools varied in a statistically, significant manner. From here, the slope values were used to determine the scalable usability of a tool. It was suggested that a tool with a higher the slope value would equate to a higher rank of usability for the tool. Therefore, in accordance with the quantitative results, the slope values for each tool were applied to this scaleable attribution of usability. However, it was imperative that the qualitative results be examined in order to confirm our quantitative findings. 310 Australian Journal of Educational Technology, 2000, 16(3) Qualitative analysis The qualitative analysis for this study was heavily weighted on the user comments gathered from the questionnaire, in respect to each of the WebCT tools. After gathering all of the user comments, our scalable measurement of usability was applied in respect to the previous usability principles defined by Shneiderman (1986), Norman (1988), Nielsen (1993), and Shackel (1991). As a result, each of the user comments were categorised respectively. For example users who commented that they were completely lost using the tool were placed in the usability category of cognitive-load (negative). On the other hand, users who commented that the tool allowed them room for creativity were placed in the usability category of flexibility (positive). After all of the comments were categorised for each of the six WebCT tools, according to positive and negative statements of usability, a distinct degree of usability for each tool became evident. It was clear that certain tools were perceived as very usable in comparison to others. These finding could now be organised on a usability scale. Therefore, the qualitative results from the study offer the following findings concerning the degree of usability for each of the six WebCT tools (Table 3): Table 3: The qualitative analysis labelled the degree of usability for each of the six tools. Tool title Degree of usability Tool 1 Customise Homepage Low usability Tool 2 Icons High usability Tool 3 Course Settings Very low usability Tool 4 Quizzes Extremely low usability Tool 5 Bulletin Board Extremely high usability Tool 6 Course Calendar Very high usability Measure Combining all of the results, each of the tools could be further represented both quantitatively and qualitatively, in respect to usability. To demonstrate our results, a pyramid (see Figure 1) was constructed, arranging each tool in a hierarchical order according to their respective quantitative slope values and their qualitative measure of usability. Interestingly, our findings suggest that the slope values agreed with each appropriate degree of usability for each of the six WebCT tools. Piguet and Peraya 311 Figure 1: The pyramid of tools combined both the quantitative and qualitative results to show the degree of usability of each of the tools. Conclusions In conclusion, several points can be concluded from the results in this study, relating to the six WebCT authoring tools. First of all, our quantitative results point out that for each of the six WebCT tools chosen in this study, perceptions of satisfaction and control varied between course designers. This finding suggests that each individual WebCT tool can 312 Australian Journal of Educational Technology, 2000, 16(3) collectively influence a course designer’s overall authoring experience, either negatively or positively. Secondly, there is a relative scalable measure of usability that can be applied to each WebCT tool. These findings were flavoured by our qualitative results, demonstrating that user comments can be categorised in two methods: 1) by usability principles defined by HCI specialists and 2) by positive and negative comments regarding these principles. These findings also suggests that each individual WebCT tool can collectively influence a course designer’s overall authoring experience, either negatively or positively. Finally, from our combined quantitative and qualitative results, our findings suggest that perception of control and perception of satisfaction are correlated with the perception of usability for six specific WebCT authoring tools. This phenomenon was demonstrated in our pyramid of tools. These results offer the idea that each individual authoring tool is important for determining the outcome of the course designer’s overall experience using WebCT. With this in mind, a course designer who uses several authoring tools with high usability will most likely have a positive experience when creating his/her web-integrated learning environments, and therefore continue to use WebCT. On the other hand, the use of several authoring tools with low usability could inflict a negative experience for the course designer, possibly leading to the discontinuation of WebCT. Therefore, it is important for authoring tools to provide the highest level of usability for its users. Overall, these results answered our previous questions regarding the six WebCT tools and were important to us for two reasons: 1) to have a better perspective of the usability of the WebCT authoring environment, and 2) to provide us with a further research basis in order to suggest usability improvements for specific WebCT authoring tools. Discussion Unfortunately, not all of the WebCT tools were tested in this study. Therefore, these findings can not be applied to the complete set of authoring tools in WebCT. However, the research in this study opens the door for others who wish to continue testing additional WebCT tools, or even other authoring environments. Piguet and Peraya 313 The usability measures proposed in this study relate to only two main concepts of HCI, satisfaction and control, while targeting only six tools. However, it can not be excluded that there are multidimensional and diverse angles to approach usability. It can be considered that different measures of usability can be constructed in different ways, each influencing a different result. With this in mind, this study tends to lead us in the direction that when satisfaction and control are measured together, a decent scale of usability can be applied to a WebCT authoring tool. Taking this in to account, it would be valuable to extend the development of this study to include examination based on the aspect of user experience when using authoring tools. More precisely, the investigation could include users' perceptions based on novice or expert experience using these tools. Additionally, it would be equally interesting to test different authoring environments, similar to WebCT, and conduct the same experiment using similar authoring tools. These findings would perhaps provide a more lucent explanation on the influences of "perception." References Ajzen, I. & Fishbein, M. (1980). Understanding Attitudes and Predicting Social Behavior, Prentice-Hall, Englewood Cliffs, NJ. Becker, D. A. A. and Dwyer, M. M. (1994). Using hypermedia to provide learner control. Journal of Educational Multimedia and Hypermedia, 3(2), 155-172. Chau, P. Y. K. (1996). An empirical assessment of a modified technology acceptance model. Journal of MIS, 13(2, Fall), 185-204. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3, Fall), 319-339. Doll, W.J. & Torkzadeh, G. (1988). The measurement of end-user computing satisfaction. MIS Quarterly, 12(2), 259-274. Fishbein, M. and Ajzen, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Addison-Wesley, MA. Garrison, D. R. & Baynton, M. (1987). Beyond independence in distance education: The concept of control. The American Journal of Distance Education, 1(3), 3-15. Lewis, J. (1995). IBM Computer usability satisfaction questionnaire: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57-78. 314 Australian Journal of Educational Technology, 2000, 16(3) Matell, M. S. & Jacoby, J. (1992). Is there an optimal number of alternatives for Likert scale items? Study 1: Reliability and validity. Educational and Psychological Measurements, 31, 657-674. Nielsen, J. (1993). Usability Engineering. Academic Press Inc., Boston. Norman, D. A. (1988). The Psychology of Everyday Things. Basic Books, New York. Nunnally, J. C. (1978). Psychometric Theory. New York: McGraw-Hill. Preece, Jenny, et al. (1994). Human-Computer Interaction. Addison-Wesley Publishing Company, 1994. Reeves, T. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer Based Instruction, 20(2), 39-46. Shackel, B. (1991). Usability – context, framework, design and evaluation. In Shackel, B. and Richardson, S. (Eds), Human Factors for Informatics Usability. Cambridge University Press, Cambridge, 21-38. Shneiderman, B. (1986). Designing the user interface: Strategies for effective human- computer interaction. Addison-Wesley, Reading MA. Wilson, B. G. (1995). Metaphors for instruction: Why we talk about learning environments. Educational Technology, 35 (5), 25-30. Worcester, R. M. & Burns, T.R. (1975). A statistical examination of the relative precision of verbal scales. Journal of Market Research Society, 17(3), 181-197. Allison Piguet is a PhD student in the Department of Educational Technology at the University of Geneva, Switzerland. She received her BA and teaching degree from San Diego State University and a Master's Degree in Educational Technology from the University of Geneva. Her research focuses on virtual campuses, authoring tools, and the development of pedagogical activities for virtual learning environments . Address: TEFCA- FPSE - University of Geneva, 40 bd du Pont d'Arve, 1205 Geneva, Switzerland. Dr. Daniel Peraya is an Assistant Professor in the Department of Educational Technology at the University of Geneva, Switzerland. His research examines computer mediated communication and web-based integrated learning environments. Address: TEFCA-FPSE - University of Geneva, 40 bd du Pont d'Arve, 1205 Geneva, Switzerland. TECFA (http://tecfa.unige.ch) is an Educational Technology Department within the Faculty of Psychology and Educational Sciences, University of Geneva.