Microsoft Word - [4]235-4440-6-CE_MM_2.doc Australasian Journal of Educational Technology, 2014, 30(1). ascilite 45 Adaptation criteria for the personalised delivery of learning materials: A multi-stage empirical investigation Stefan Thalmann Innsbruck University School of Management, University of Innsbruck, Austria. Personalised e-Learning represents a major step-change from the one-size-fits-all approach of traditional learning platforms to a more customised and interactive provision of learning materials. Adaptive learning can support the learning process by tailoring learning materials to individual needs. However, this requires the initial preparation of content upfront, which is a laborious task – and organizations have to target their limited resources effectively. In order to guide the process of creating adaptive learning materials, the criteria for adaptation – or adaptation needs – have to be known. The aim of this paper is to identify these adaptation criteria, applying a mixed method procedure. First, thirty adaptive systems selected from the literature are investigated using a qualitative content analysis. Then, the resulting set of adaptation criteria is validated by experts in the form of a series of two online questionnaires. As a result, a set of 13 adaptation criteria representing different adaptation needs emerge. Introduction Personalised learning content has been shown to increase learner interest, comprehension and hence their learning success (Triantafillou, Pomportsis, Demetriadis, & Georgiadou, 2004). The personalisation of learning material in the form of a content adaptation tailored to the needs of the learner is frequently proposed as one of the ways by which the acceptance and efficiency of e-learning can be increased (Brusilovsky, 2003; Chen, Lee, & Chen, 2005; Cristea, 2004; Gkatzidou & Pearson, 2009). Furthermore, the rise of mobile learning increased both the potential and the demand for the adapted delivery of learning materials (Chen, Chang, & Yen, 2012). Research has been conducted on the technical realisation of adaptive e-learning, and led to the development of a number of research prototypes (e.g. Conejo, Guzmán, Millán, Trella, Pérez-De-La-Cruz, & Ríos, 2004; Kayama & Okamoto, 2001; Maier, Armstrong, Hall, & Ng, 2005). However, one major challenge remains, which is the creation of suitably prepared learning materials (Akbulut & Cardak, 2012; Cristea & Stewart, 2006b; Foss, Cristea, & Hendrix, 2010). The adaptive provision of learning materials involves the identification of content that is relevant to the learner (Bunt, Carenini, & Conati, 2007). To this effect, user preferences and context must be known and represented in a way that is appropriate to adaptive systems. Numerous approaches that attempt to categorise people according to differences in learning and cognitive styles are known (see Coffield, Moseley, Hall, & Ecclestone, 2004). Also many attempts focussing on dimensions representing the learner context (see Zimmermann, Specht, & Lorenz, 2005) can be found in the literature. Together, these categorisation approaches structure and facilitate the authoring of (peronalised) educational resources. In this way, authors first identify an adaptation need for a concrete type of learner, acting in a specific context (see Figure 1) and then the learning materials suited to this particular adaptation need (e.g. adapted to language) are created. This emphasises the need to ensure that the created content also reflects the specifics of the adaptation need, (Foss et al., 2010; Yalcinalp & Gulbahar, 2010) which are described in the associated metadata (i.e. changed duration or language). From a pedagogical perspective, this task is guided by existing structuring models and theories. However, such guidance is missing from a technological perspective representing the characteristics of adaptive systems. The central question here is: Can the adaptation need be detected by adaptive systems and linked to the characteristics of people and their context? Authors need clarification on this question to ensure that the modified learning material can be delivered appropriately. A categorisation approach of these technical features would help designers with defining the required preparations to tailor them to learning styles or contextual characteristics. Further, it helps to increase the standardised description of such instructional materials and thus facilitates the content exchange across organisations. Akbulut and Cardak (2012) therefore demand a further investigation and also the standardisation of these factors to eliminate obstacles in creating contents for adaptive educational systems. Australasian Journal of Educational Technology, 2014, 30(1). 46 learner e.g.  learning  and  cognitive  styles learning  material context e.g.  location  or  work  task adaptation need su its suits characteristics   diffent  to  the   original  version   (meta  data) duration language contained media preparation e.g.  adapt  to   language adaptive  system can  be  de tec ted Figure 1. Relationship between adaptation need and learning materials The standardisation of these factors can be achieved through a categorisation of members who share common properties, and with a set of rules which identifies exclusive members of a category (Doty & Glick, 1994). Criteria provide descriptions on which a judgment or decision can be based, hence enabling decisions about allocating membership to a category. In the following, the criteria that will allow such a classification are referred to as adaptation criteria. Adaptation criteria describe the characteristics of an adaptation need on a conceptual level. They represent aspects that can guide the preparation of learning materials and to which learning materials can be delivered adaptively. The underlying research question of this paper is: To which set of (adaptation) criteria can contents be adapted? The aims of this paper are, (1) to propose a set of adaptation criteria based on the existing technology, and (2) to refine and validate this initial set of adaptation criteria with experts from the field. We start by discussing related work and the research design. Then we present the establishment of the set of adaptation criteria and the refinement and validation of criteria in two empirical studies, followed by a discussion of results and concluding with an outlook on further research. Related work Adaptive educational systems are a subclass of virtual learning environments (Mueller & Strohmeier, 2011) intended to provide peronalised e-learning. According to the frequently cited definition of Oppermann (1994) “a system is called adaptive if it is able to change its own characteristics automatically according to the user’s needs” (p. 456). Paramythis and Stephanidis (2005) describe the characteristics and the user needs more concretely as “the given attributes of users and their particular context of use” (p. 80). The definition of given attributes is essential because changes of behaviour and contextual variables have to be defined during implementation. Predefined behaviour has been represented in different models, such as, for example, the adaptation model, the domain model, the internal model and the user model (Brusilovsky, 2001). The key feature of adaptive systems delivering learning materials is their ability to perform content adaptation by translating an identified user need into an adaptation action, which primarily entails the selection of learning material (Bunt et al., 2007). Thus, suitable learning objects have to be available for each identified user need, and for each adaptation need. However, the continuum of adaptation needs addressed by adaptive systems and their match to suitable learning material, have received little attention so far, as research has primarily concentrated on learning styles rather than on other factors (Akbulut & Cardak, 2012). To facilitate the authoring of adaptive learning material, however, these factors ought to be known so that they can be integrated into authoring systems to build the basis of analytical and prescriptive models and to prompt and tutor instructional designers as recommended by O’Neil (2008). Australasian Journal of Educational Technology, 2014, 30(1). 47 Authoring learning materials for adaptive systems requires a modularisation of contents. Hence, learning materials need to be divided into bite-size pieces that can be reused and repurposed flexibly. These atomic containers are called 'learning objects' (Boyle, 2003). Metadata describe the characteristics of learning objects, which are relevant for selection within adaptive systems and thus are of particular importance (Pawlowski & Bick, 2006). The standardisation of metadata is needed to enable the effective exchange and reuse of learning objects across organisations (Schoonenboom, 2012). Such standards for metadata are established in order to provide a standardised set of data elements (Maier & Thalmann, 2007). The most widespread metadata standard in the field of learning technology is IEEE learning object metadata (LOM) (Klemke, Ternier, Kalz, & Specht, 2010). Previous research on learning objects has been conducted on their technical realisation (e.g Ochoa, Cardinaels, Meire, & Duval, 2005; Zouaq, Nkambou, & Frasson, 2007), on their annotation with suitable metadata (e.g Brooks & McCalla, 2006; Duval, Hodgens, Sutton, & Weibel, 2002), and on their pedagogical design and adaptive composition (e.g Conclan, Dagger, & Wade, 2002; Karampiperis & Sampson, 2006). Research on adaptation criteria is, however, scarce. The initial literature review brought together hitherto scattered statements describing the different aspects which adaptive systems consider for adapting contents (see for example, Brusilovsky, 1996, 2001; Cristea, 2004; Gkatzidou & Pearson, 2009; Kay, 2000; Kobsa, Koenemann, & Pohl, 2001; Pazzani & Billsus, 2007). However, none of these papers paid attention to the categorisation of these aspects or to the preparation of content. The statements differ in the level of detail provided and focus only on the respective perspectives of specific adaptive systems. A more comprehensive presentation can be found in Triantafillou, Georgiadou and Economides (2007), where aspects from five different papers (including two from the same author) are reviewed and described. The aim of the study by Triantafillou et al. (2007) was to identify features for adaptation, which can primarily be used in user modelling and adaptive testing. In light of the broad spectrum of adaptive systems and the number of authors, an investigation which considers a larger and more diverse sample seems useful. Hence, the main objectives of this paper, following on from Triantafillou et al. (2007), are to investigate in greater detail, and empirically, the adaptation needs useful for informing the adaptive delivery of learning objects. Research design A combination of qualitative and quantitative research methods were used to identify and validate adaptation criteria. In the absence of a testable hypothesis, a multi-method, or mixed method approach was deemed appropriate (Mingers, 2000); and a sequential design was chosen, starting with a qualitative explorative content analysis and followed by confirmatory quantitative surveys (Hanson, Creswell, Creswell, Plano Clark & Petska, 2005). Figure 2 presents an overview of the research design. Australasian Journal of Educational Technology, 2014, 30(1). 48 survey with developers of all identified systems revised set of adaptation criteria identification of adaptive systems structured content analysis of a stratified sample survey with developers of investigated systems initial set of adaptation criteria final set of adaptation criteria with importance and effort rating list of adaptive systems 1st phase: establish the initial set of adaptation criteria 2nd phase: validate the interpretations from the 1st part 3rd phase: validate the set of adaptation criteria and to collect ratings on importance and effort Figure 2. Overview of the study design The first phase of the study involved an initial literature review which identified a multitude of adaptive systems, all of which were research prototypes (e.g. Brusilovsky, 1996, 2001; Gkatzidou & Pearson, 2009; Kay, 2000; Kobsa et al., 2001; Pazzani & Billsus, 2007). These adaptive systems seemed to provide a sound basis for establishing a set of adaptation criteria, and it was decided therefore to investigate existing adaptive systems. The scientific papers on adaptive systems identified in this way contained descriptions of their adaptive behaviour which could then be drawn on for the categorisation. It was also deemed that peer-reviewed scientific papers would by definition be able to offer suitable sources as a starting point for this study. Due to the fact that a comprehensive initial set of adaptation criteria was not available, a research method suitable for establishing a set of adaptation criteria was needed. A qualitative content analysis seemed the most appropriate approach for this purpose. Content analysis is a form of qualitative analysis suitable for analysing large amounts of textual information and to extract information about texts. The analysis itself is typically performed by coding the raw data (Patton, 2002). In contrast to purely inductive approaches, content analysis offers the option of using initial categories as a kind of structuring element. These categories are usually identified from the literature and form the starting point of the study (Patton, 2002). These initial categories, however, can be adjusted, as new insights emerge during analysis. Thus, data analysis is guided by both theoretical criteria and, at the same time, it is open to new discoveries. The second part of the study builds on the adaptation criteria established in the content analysis. To ensure the validity of the results that have been gained by analysing scientific papers, it is necessary to validate the qualitative results (Silverman, 1993). Quantitative methods can be used to test the results derived from qualitative research and based on the researcher's understanding (Lee, 1991). The first validation aimed to avoid misinterpretations occurring during the content analysis, and hence, the authors whose publications on adaptive systems featured in the content analysis were contacted. Additionally, the authors were requested to rate the consistency of the intial set of adaptation criteria. This involved an online questionnaire focussing directly on each of the 30 adaptive systems investigated. Australasian Journal of Educational Technology, 2014, 30(1). 49 After validating the interpretations of the papers used in the content analysis, the third part of the study aims at validating the criteria established in general with a larger sample. For this purpose it seemed appropriate to additionally involve the authors of papers on adaptive systems who were not contacted for the content analysis. Hence, a second questionnaire-based survey targeted all the identified authors of papers on adaptive systems. The next section sets out how the three parts of the study were performed. Phase one: Establishing the initial set of adaptation criteria A wide variety of descriptions of adaptive systems are available. However, not all sources are deemed trustworthy - notably websites or advertisement flyers. Academic publications, on the other hand, provide a certain degree of robustness due to being traceable if not to a review board, at least to an academic organisation. Thus, only academic papers, such as conference papers, journal articles, technical reports or Ph.D theses were considered for the study. The goal of the following content analysis was: To analyze existing adaptive systems and to develop a set of adaptation criteria, representing adaptation needs to which learning materials can be prepared and delivered adaptively. Procedure The literature review identified 158 adaptive systems. Only adaptive systems that had been implemented, and subsequently described in academic publications were considered. To cluster the sample, the well- recognised typology of adaptive hypermedia systems proposed by Brusilovsky (2001) was adopted, with the following categories of systems used, respectively, for education, for serving on-line information and for information retrieval problems. As presented below, all 158 systems identified could be assigned to one of these categories: • 56 adaptive educational systems • 52 adaptive on-line information systems • 50 adaptive systems for information retrieval. Within the initial literature review and the search for adaptive systems a great number of similarities of adaptive systems within any one category can be observed. Due to the fact that two validation cycles were planned, a stratified sample of 30 systems – ten per category – seemed sufficient to identify the main characteristics of each category. The systems in each category are ordered alphabetically and the first and fifth system of each category are chosen for the investigation. The aim was to find two academic papers for every system in order to arrive at a more diverse perspective on the adaptive system. However, this was not possible to achieve for all selected systems, as for four systems only one academic paper could be found. The paper had to provide a description of the system researched and the system’s adaptive behaviour. Overall, 56 academic papers were considered for the detailed investigation of adaptive behaviour. For each adaptation criterion a short description, an assignment rule and a list of examples was created by applying the following procedure. First, the parts of the research paper (extracts) which described how contents can be delivered adaptively are highlighted. For example, in paper 3 “[…] provides adaptation to various learning styles” or “[…] take into account the user’s preferences for certain types of media” are typical phrases. If this aspect fullfills the assignment rules of an existing adaptation criterion, the text is labelled (coded) with the name of the criterion. If the aspects fit an existing adaptation criterion but the assignments rules do not match exactly, the description and the rules of the adaptation criterion are revised. This required a re-checking of all previous assignments of this adaptation criterion. Otherwise, a new adaptation criterion was established. Additionally, consolidation cycles were necessary in which preliminary adaptation criteria were split or merged to ensure a homogeneous categorisation on a similar level of detail. Results The result of the structured content analysis was a set of the following 13 adaptation criteria: content preferences, bandwidth, device requirements, knowledge structure, language, learning style, location, preferences for media types, presentation preferences, previous knowledge, user history, user request and user status. This initial set of adaptation criteria is subject for discussion and will be changed according to Australasian Journal of Educational Technology, 2014, 30(1). 50 the feedback gathered. For this reason this intermediate result is not explained in detail here, but instead, the final set of adaptation criteria are presented in detail after phase four. Phase two: Validation with developers of investigated systems The limitations that came to light in the first part of the study raised some doubts over the reliability of the results: First, the criteria are established on the basis of the authors’ interpretations, and neither misinterpretations nor a limited or biased perspective can be excluded. Second, a complete description of system functionality cannot be guaranteed, although for most systems two papers were selected. Third, the papers are no longer up-to-date due to lengthy review and publishing procedures. Fourth, the sample of 30 adaptive systems may not represent the most important aspects. Fifth, the assignment of categories is probably imperfect, and complete coverage of all aspects cannot be guaranteed. The main goal of this second phase is to overcome these limitations by asking developers to check the assigned ratings of their systems and thus to detect misinterpretations. Additionally, the developers were asked for completeness, e.g. for additional criteria and integrity of the presented set of criteria. The intention is to give developers the opportunity to correct values for their systems, to comment on the proposed criteria and to report any missing criteria. Procedure A questionnaire, by means of an online survey tool, was used to achieve the aim. The target group comprised the combined authors of the papers considered for the structured content analysis. Authors were contacted in order to evaluate their system regarding the established criteria. Overall, 74 authors were contacted via e-mail - at least one person for every system. A reminder was emailed to 11 authors of seven systems who had not responded by the deadline. The survey requested authors to check assignments of identified adaptation criteria in their system. Each criterion was described with a short explanation and with one example. A check box with “[name of the system] supports this” and “[name of the system] does not support this” was assigned for each criterion. The ratings from the structured content analysis were presented as default values of the check boxes in the questionnaire. They could be changed by clicking on a tick box, and new adaptation criteria or comments and suggestions could be entered into a text box. Presenting the results of the structured content analysis as default values had the advantage of creating a greater sense of ownership in these ratings and motivated people to take part in the survey. Furthermore, the positive and direct feedback leads to the assumption that the authors contacted understood these ratings as direct feedback on their work. Optional textboxes allowed authors to enter their own proposals for criteria, if applicable, and add additional comments. Results The survey evaluated 23 adaptive systems or 77% of the sample. Calculating the return rate for the authors is questionable, because some authors completed the questionnaire together or they delegated the task to another person in the group. Participants also made 22 new assignments from criteria regarding systems. The author rechecked the papers used for the structured content analysis for any evidence of these new assignments but could not find any. It is probable that either these features were not described in these papers or they had been developed in the meantime. Only one assignment which was set by the author was removed by a survey participant. The author compared this with the papers used for the structured content analysis and found that the assignment could be seen as borderline in the yes or no rating. The rating for this particular system was therefore removed. Summarised results are depicted in Figure 3, showing criteria on the y-axis and the number of systems which fulfil the criteria on the x-axis. Grey bars represent the frequency determined in the content analysis and black bars the frequency after the survey. The criteria are sorted by frequency from the survey performed and only minor changes in the ranking between both results can be observed. Australasian Journal of Educational Technology, 2014, 30(1). 51 0 5 10 15 20 25 30 user  status bandwidth language device  requirements preferences  for  media  types learning  style location presentation  preferences user  request previous  knowledge knowledge  structure user  history content  preferences occurence  from  survey occurence  from  content  analysis Figure 3. Frequency of criteria after the first survey The first four highest ranked criteria (content preferences, user history, knowledge structure and previous knowledge) have the highest frequency and occur in more than 60% of the systems. It is notable that content preferences, which occur in 90% of the systems, appear to be a very common aspect within adaptive systems. An analysis of systems and criteria over time showed no obvious trend. Therefore, neither the number of systems, which fulfil certain adaptation criteria, nor the total number of adaptation criteria that had been fulfilled by the systems, saw an increase over the years. The survey provided not only ratings of criteria but also generated comments, which were in the majority positive about the study and the research topic itself. Many participants commented on their decisions for assignments or made general remarks. Five participants made suggestions for new categories or proposed new aspects, which might not have been covered by the current criteria. All five remarks and suggestions were carefully checked by applying the following procedure: in relation to the adaptation criterion concerned, if it was already covered by the assignment rules and the descriptions, no changes were performed. This was the case for four remarks. If the aspect was not covered, the adaptation criterion was revised accordingly. This was the case for the proposed criterion “User-defined Content Source”. From the authors’ point of view this is a sub-aspect of a content preference. As this aspect was not covered by the description and rule of the adaptation criterion “content preference” this criterion was adapted accordingly. There was no instance of where a remark or suggestion could not be assigned to an existing adaptation criterion. As a result, one adaptation criterion was revised, no new adaptation criteria were added and no existing ones deleted. Phase three: Validation with developers of all identified systems The main goal of the second phase was to detect misinterpretations in the content analysis by passing them by the scrutiny of the system developers of the investigated adaptive systems. However, it seemed more appropriate to use a larger sample for the validation of the proposed adaptation criteria in general. Hence, developers of the 128 systems that were not investigated in the content analysis were invited to contribute their perspective to the study. The aim of phase three is to revise the 13 adaptation criteria using a larger group of experts and to evaluate the importance of and the effort involved in the implementation of criteria. Australasian Journal of Educational Technology, 2014, 30(1). 52 Procedure This study does not focus on the evaluation of specific systems, but on the in-depth evaluation of the identified adaptation criteria. Thus, all authors of the 158 identified adaptive systems could be considered for the second survey. It can be assumed that people who wrote a scientific paper about an adaptive system have a substantial amount of knowledge and can be rated as experts in this field. Overall, 238 persons were approached. All authors were contacted via a personalised e-mail that contained the name of the person, a short description of the study goals and a link to the questionnaire. The participants were requested to rate the usefulness and effort involved in the creation of contents regarding each criterion. The answers had to be given within the scope of a visual analog scale with a range from 0 to 100 (Wee, Fong, Tse, Machin, Cheung, Luo & Thumboo, 2008), allowing a midway rating. Visual analog scales are continuous scales in which respondents can interactively slide a marker to the position of the scale that best fits their rating (Dillmann, Smyth & Christian, 2009) thus offering the avantage of a more subtle and differentiated rating. This method also presents a higher level of reproducibility and sensitivity than Likert and Borg scales (Grant, Aitchison, Henderson, Christie, Zare, McMurray & Dargie, 1999). Rating usefulness and effort is quite difficult and needs a reference point for exact ratings. Because of the great variety of adaptive systems and the general purpose of the adaptation criteria, the specific reference points cannot be given. Hence, the focus was on an estimation of usefulness and effort for the adaptation criteria rather than an exact measurement in a specific scenario. Asking participants to rate usefulness and effort would trigger a reflection in participants on, respectively, the benefits and effectiveness of the criteria, and their technical implementation. The online questionnaire contained two additional text boxes, in which people could add proposals for adaptation criteria and feedback on the proposed set of adaptation criteria. The survey was realised in form of an online questionnaire for all the experts contacted. Results At the end of the survey period 51 questionnaires were completed (out of a possible 238 – a response rate of 21.4%). One of the respondents rated effort and usefulness for all criteria with the maximum value (100). These cases should be eliminated, as they do not represent valid data (Osborne & Overbay, 2004). Consequently, this data record was eliminated and the remaining 50 data records are considered in the following. Descriptive statistics representing the results for usefulness are presented in table 1, and results for effort are presented in Table 2. The criteria are ranked according to their mean value, calculated from the responses in both cases. As expected, both rankings differ and a large spread of the mean value can be observed. Large differences in the mean value between the first ranked criterion and the last ranked criterion can also be observed for both effort and usefulness. The maximum entered value of all criteria concerning usefulness and effort is at least 95. The minimum entered value is three times 0 for usefulness and seven times 0 for effort. Thus, it can be observed that the response is wide-spread. The high variation of results, which is indicated by a comparably high standard deviation, leads to the assumption that the validity of the ranking is questionable. However, the results can be seen as an informal informative basis for further studies. In addition to the rating of criteria, the survey generated a number of comments. Some participants made suggestions for new categories or proposed aspects, which might not have been covered by the current criteria. All seven remarks and suggestions were carefully considered by applying the following procedure: The remark or suggestion was checked carefully for the affected adaptation criterion. If it was already covered by the assignment rules and the descriptions, no changes were performed. This was the case for three remarks. If the aspect was not covered, the adaptation criteria was revised accordingly. This was the case for the proposed criterion “user status”. One study participant recommended that the relation between user-related characteristics and the impact on the ability to absorb knowledge should be considered. Another study participant recommended that this aspect be considered from a psychological viewpoint, as cognitive load. These valuable comments were taken on board and the criteria revised Australasian Journal of Educational Technology, 2014, 30(1). 53 accordingly. Furthermore, it was recommended to add trust to the definition of content preferences to complement aversions and preferences. This comment was also considered and the definition revised accordingly. Finally, the consideration of cognitive abilities which are distinct to learning styles as they describe the user’s ability to receive contents in addition to the learning style was recommended. The user’s ability to receive contents is influenced by his or her cognitive abilities, i.e. working memory capacity, inductive reasoning ability and associative learning skills, and the cognitive styles that are associated with these characteristics. The consequence for an adaptive delivery of contents is a didactical revision of the contents so that they suit the learner needs. Hence, in addition to the learning style, a more detailed investigation of a learner’s characteristics in regard to a didactical preparation is performed and the criterion ‘learning style’ was renamed as ‘didactical approach’ and the definition revised accordingly. Hence, one adaptation criterion was revised, no new criteria were added and no existing ones deleted. Table 1. Descriptive statistics on usefulness Criterion M Min Max SD learning style 70.8 7 100 24.0 knowledge structure 67.5 0 99 22.7 previous knowledge 64.8 21 100 21.7 content preferences 64.2 13 100 24.1 user status 59.8 4 99 27.6 language 55.7 0 100 33.0 user history 54.8 4 100 26.8 device requirements 54.2 2 100 27.0 location 52.3 7 100 26.0 user request 51.0 4 100 29.9 pref. for media types 41.6 2 100 27.5 bandwidth 41.8 5 100 26.0 presentation preferences 36.7 0 98 26.1 Table 2. Descriptive data on effort Criterion M Min Max SD learning style 80.9 19 100 19.3 user history 76.8 32 100 19.5 previous knowledge 70.4 12 100 23.9 content preferences 67.9 0 100 26.3 language 66.9 0 100 28.2 learning style 63.7 0 100 31.1 user request 60.4 7 100 25.6 device requirements 60.7 7 100 25.7 location 58.5 0 99 31.6 presentation preferences 52.2 0 95 26.1 bandwidth 49.8 1 100 27.4 user status 48.9 0 99 28.0 pref. for media types 42.4 0 99 27.8 Australasian Journal of Educational Technology, 2014, 30(1). 54 The Final Set of Adaptation Criteria In the following the final set of adaptation criteria is presented in Table 3. Criteria are ordered alphabetically and each is given a short description and an example. Table 3. Final Set of Adaptation Criteria Name of criterion & description Example Bandwidth: Data transfer rate that is available during a session with the system. A user has a slow Internet connection and thus receives only versions of the content suitable for this bandwidth. Content preferences: Preferences, trust and aversions for presented contents or content sources. A user has a content preference for data bases and is given knowledge elements dealing with data bases. Device requirements: Technical characteristics of hardware and basic software that are relevant for accessing the system. A user accesses the system with a mobile device, having a small screen resolution. Thus, only versions that fit that resolution will be delivered. Didactical approach: Methods considering users’ preferences and cognitive abilities during learning. A user could be classified as an “implementer” (according to Jackson’s Learning Styles, see (Coffield et al., 2004)) and consequently this user is given resources that are enriched with practical examples. Knowledge structure: A list of terms or taxonomy of concepts, sub-concepts and their relations in a knowledge domain. It is defined that SQL is a sub-topic of database and a user with a content preference in databases receives also documents related to SQL. Language: Ability or preference of a user for the language that is used for content delivery. A user speaks only Spanish – she thus is given only the Spanish language version of the knowledge element. Location: Physical coordinates which can be related to pieces of content. A user stands in front of the painting of Mona Lisa and receives knowledge elements that are dealing with the picture of Mona Lisa. Preferences for media types: Preferences for the technical format of contents. A user likes to receive PDF documents – she thus only receives PDF versions of content. Presentation preferences: Preferences for the style in which contents are delivered. A user prefers very large fonts and red colours. The content is thus rendered with large fonts and a red- coloured layout. Previous knowledge: Knowledge of the user, acquired in the past and relevant for using the system, which has to be considered for the information provision. A user has a technical background and good knowledge of jargon. The user thus gets a version of a technical report without explanations of the basic terms. User history: A collection of data describing previous interactions with an adaptive system on an individual or a group level. The system tracked the user opening only knowledge elements with the status “final”. It then recommended only knowledge elements with this status to the user. User request: Additional pre-defined interaction options for user-initiated adaptation. The user selects 10 minutes from a drop down menu, as his or her available time frame for interacting with the system provides only resources which require a maximum of 10 minutes’ interaction. User status: User-related or environmental characteristics describing the user’s current activities, cognitive load or stress level and their related impact on the user’s ability to absorb knowledge. A user defines her personal status as tired and thus receives only versions of the content which do not require high concentration. Australasian Journal of Educational Technology, 2014, 30(1). 55 Discussion Thirteen adaptation criteria representing adaptation needs to which contents can be prepared are the key outcome of this paper. These adaptation criteria can facilitate the design and creation of contents, the pedagogical planning, technological implementation, and the organisational scaling of adaptive learning technologies. In the following the implications that the adaptation criteria have for the design, organisational, technical and pedagogical aspects are discussed. The facilitation of the design and creation of adaptive contents is crucial to avoid ill-prepared contents, which are a major barrier for the acceptance of adaptive learning technology (Akbulut & Cardak, 2012; Cristea & Stewart, 2006a). Adaptation criteria can thus support the planning of the content creation process by providing a classification into which similar preparation tasks can be pooled, particularly where the learning objects are to be used adaptively. Akbulut and Cardak (2012) anticipate that such a standardised structure will help to eliminate obstacles in creating content for adaptive educational systems. More specifically, adaptation criteria could be integrated into authoring systems. Here, they can form the basis of analytical and prescriptive models for instructional and adaptive design and they could be used for prompting and tutoring the instructional designer as recommended by O’Neil (2008). A standardised tool kit for each adaptation criterion in an authoring environment could thus help increase the efficiency of preparing the content design and creation, and thus, reduce the input required by human effort. The results of phase three of the study indicate that the usefulness, especially of those adaptation criteria which demand primary manual preparations, like didactical approach or previous knowledge, is rated very highly. Additionally, the ratings for the preparation effort of these adaptation criteria are similarly high. By contrast, both the usefulness and the required preparation effort for criteria with a high automatisation potential, like preferences for media types or bandwidth, are comparably low. Furthermore, the estimation of effort can also benefit from such a standardised grouping of preparation tasks, and the estimation of effort collected in phase three of the study represents a promising starting point in this respect. The organisational scaling of learning technologies, especially across organizations, is important for their success. This would particulary facilitate the reusability of content and cost savings as well as an extension of available content (Boll, 2003). Authoring contents for adaptive usage is currently very costly, as authoring systems at this point are at an immature phase of development in terms of productivity versus cost (O’Neil, 2008). Large cost saving effects could be expected from standardising the preparation tasks. Modularisation would be another important driver, particulary across organisations, which, however, requires a certain extent of standardisation. The adaptation criteria therefore provide a classification into which similar preparation tasks can be pooled and the results of the phase three of the study provide an appreciation of their expected usefulness and effort. Boyle (2003) proposes to perform the modular engineering of learning objects according to different dimensions in order to achieve their repurposability. These dimensions could be represented by the proposed set of adaptation criteria, particularly where the learning objects are to be used adaptively. Additionally, the adaptation criteria represent a valuable basis for standardising both development and evaluation processes across organisations. The adaptation criteria could also be used as selection criteria in choosing and deploying adaptive systems as well as authoring systems at the organisational level. Technical implications can be expected especially in the standardisation of learning object descriptions. The modularisation and reuse of learning objects across organisations both require a standardised description, which is technically realized by metadata (Pawlowski & Bick, 2006). The characteristics of learning objects are described by their metadata, using the elements ‘keyword’, ‘subject’ or ‘educational’, from LOM, for example. Even if a variety of further metadata standards and specifications for educational material can be found, such as IMS learning design, to the author’s knowledge, the suitability of an adaptive purpose is not explicitly considered in any one these standards or specifications. The characteristics of a learning object affected by an adaptive preparation are represented by different metadata elements. Even if all changed characteristics could be represented by current metadata standards or specifications, the variety of changed characteristics would be vast. Hence, it seems difficult to detect the overall purpose of the changes in comparison to the non-prepared or original version of the learning object. With overall purpose, the meaning of adaptation needs to be addressed, which would assist with planning curricula or technical implementations. The proposed adaptation criteria can be used as metadata elements representing adaptation needs. For this purpose, application profiles (Baker, Dekkers, Heery, Australasian Journal of Educational Technology, 2014, 30(1). 56 Patel & Salokhe, 2001) extending existing metadata standards and specifications, preferably LOM, seem promising. Pedagogical benefits can be expected since it becomes easier for personnel with less technical knowledge to plan didactically designed adaptive arrangements using adaptation criteria. In particular, if the adaptation criteria are used as basis of analytical and prescriptive models in authoring systems they could be used for prompting and tutoring the instructional designer as recommended by O’Neil (2008). The results from phase three of the study indicate that the usefulness of preparations according to learning styles are considered very high. However, the efforts needed to prepare contents accordingly are also considered high. Hence, an expected increase in the productivity of content preparation by the recommended integration into authoring systems seems particulary valuable. The present study was undertaken with a mixed method approach, following Teddlie and Tashakkori (2009). Qualitative methods were used to identify adaptation criteria, and quantitative methods to validate them. Combining different methods for one research problem can help to overcome the common method bias (Podsakoff, MacKenzie, Lee & Podsakoff, 2003). One limitation of the performed structured content analysis was the sequence of systems performed in the analysis. The selection was based on the three categories and ten systems from the educational category which were chosen at the outset. Following the selection, systems from the other categories were chosen consecutively. In the early stages of the coding process, the greater receptivity to the range of ideas might have led to systems from the category ‘education’ being accorded a disproportionate importance. Nevertheless, categories were adjusted during the analysis of systems from the other categories. The arising limitations were addressed by the quantitative survey, which asked experts to add missing criteria and other suggestions. Critics might argue that the content analysis ignores the varying meanings of text fragments, and that relationships with the context can be lost. Losing the context is a by-product of developing a classification and cannot be avoided in a structured content analysis. However, misinterpreting the context and losing the context during the development of the classification is problematic. For that purpose, the second study set out to mitigate this risk. The aim was to detect a misinterpretation of the context by involving the very people who knew the context of the paper and the adaptive system well. Refinements incorporating feedback from both studies were performed so that definitions of criteria were extended and broadened by the comments, to ensure they fitted the descriptions. Summary and Outlook This paper proposes a classification of adaptation needs to which content can be prepared with reference to thirteen adaptation criteria. Here, the paper makes a significant contribution to the initial research of Triantafillou et al. (2007) by considering 30 adaptive systems and by validating the set of criteria through experts, in two online surveys. The authors’ ratings for each system and the set of criteria provided the primary focus of the first survey. The second survey focused on a review of the established set of criteria and a rating of their effort and usefulness. The use of an established set of adaptation criteria for the description of learning objects could facilitate their reuse. Further, the creation of adaptive contents could be facilitated by integrating the established criteria into authoring systems. Both could lead to an increased penetration of adaptive e-learning, and to a better amortization of the preparation costs. Future research could support adapative learning further by investigating the following: from a technical point of view, research could promote the integration of the adaptation criteria in the metadata standardisation and in the standardisation of learning technologies. From a design perspective, typical instances for each adaptation criterion with a match to the existing meta data elements could be identified and proposals for the design given. From a pedagogical viewpoint, more attention could be paid to the matching of different perspectives of adaptive e-learning (represented by the adaptation criteria) to existing learning theories. Further empirical evidence about the impact of adaptive learning technologies on learning success is also needed. Finally, from an organisational point of view, the efforts associated with preparations according to the adaptation criteria should be investigated to enable more reliable planning. Additionally, adaptation criteria could be used to guide users in the co-construction of learning resources, i.e. performing adaptive versioning (Gu, Zha, Li, & Laffey, 2011). Research on incentives for each adaptation criteria to which the users should create new learning resources would seem very promising in this respect. Australasian Journal of Educational Technology, 2014, 30(1). 57 Acknowledgments The research leading to the presented results was partially funded by the European Commission under the 7th Framework Programme (FP7) and LEARNING LAYERS (contract no. 318209). References Akbulut, Y., & Cardak, C. S. (2012). Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011. Computers & Education, 58(2), 835-842. Baker, T., Dekkers, M., Heery, R., Patel, M., & Salokhe, G. (2001). What terms does your metadata use? Application profiles as machine-understandable narratives. Paper presented at the Int’l. Conf. on Dublin Core and Metadata Applications, Tokyo. Retrieved from http://dcpapers.dublincore.org/pubs/article/view/654 Boll, S. (2003). MM4U - A framework for creating personalized multimedia content. Paper presented at the Ninth International Conference on Distributed Multimedia Systems, Miami, USA. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=3C9B70A902331ED42F3DF0E0D25606DE ?doi=10.1.1.59.6880&rep=rep1&type=pdf Boyle, T. (2003). Design principles for authoring dynamic, reusable learning objects. Australian Journal of Educational Technology, 19(1), 46-58. Brooks, C., & McCalla, G. (2006). Towards flexible learning object metadata. International Journal of Continuing Engineering Education and Lifelong Learning, 16(1/2), 50-63. Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User Adapted Interaction (Special issue on adaptive hypertext and hypermedia), 6(2-3), 87-129. Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User Adapted Interaction, 11(1/2), 87- 110. Brusilovsky, P. (2003). Adaptive navigation support in educational hypermedia: The role of student knowledge level and the case for meta-adaptation. British Journal of Educational Technology, 34(4), 487-497. Bunt, A., Carenini, G., & Conati, C. (2007). Adaptive content presentation for the web. In P. Brusilovsky, A. Kobsa & W. Nejdl (Eds.), The Adaptive Web (pp. 409 – 432). Berlin/Heidelberg: Springer LNCS 4321. Chen, C., Lee, H., & Chen, Y. (2005). Personalized e-learning system using item response theory Computers & Education, 44(3), 237-255. Chen, I. J., Chang, C. C., & Yen, J. C. (2012). Effects of presentation mode on mobile language learning: A performance efficiency perspective. Australasian Journal of Educational Technology, 28(1), 122- 137. Coffield, F., Moseley, D., Hall, E., & Ecclestone, K. (2004). Learning styles and pedagogy in post-16 learning: A systematic and critical review. London: Learning and Skills Research Centre. Retrieved from http://www.hull.ac.uk/php/edskas/learning%20styles.pdf Conclan, O., Dagger, D., & Wade, V. (2002). Towards a standards-based approach to e-learning personalization using reusable learning objects. Paper presented at the World Conference on E- Learning in Corporations., Dublin, Ireland. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.6916 Australasian Journal of Educational Technology, 2014, 30(1). 58 Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-De-La-Cruz, J. L., & Ríos, A. (2004). SIETTE: A web–based tool for adaptive testing. International Journal of Artificial Intelligence in Education, 14(1), 1-33. Cristea, A., & Stewart, C. (2006a). Authoring of adaptive hypermedia. In G. D. Magoulas & S. Y. Chen (Eds.), Advances in web-based education: Personalized learning environments (pp. 225-252). Hershey, PA: IGI Global. Cristea, A. I. (2004). What can the Semantic Web do for Adaptive Educational Hypermedia? Journal of Educational Technology & Society, 7(4), 40-58. Cristea, P., & Stewart, C. (2006b). Automatic authoring of adaptive educational hypermedia. In Z. Ma (Ed.), Web-based intelligent e-learning systems: Technologies and applications (pp. 24-55). Amsterdam: IDEA. Dillmann, D. A., Smyth, J. D., & Christian, L. M. (2009). Surveys - The tailored design method (3rd ed.). Hoboken: Wiley. Doty, D. H., & Glick, W. H. (1994). Typologies as a unique form of theory building: Toward improved understanding and modeling. The Academy of Management Review, 19(2), 230-251. Duval, E., Hodgens, W. H., Sutton, S., & Weibel, S. L. (2002). Metadata principles and practicalities. D- Lib Magazine, 8(4). Foss, J. G. K., Cristea, A. I., & Hendrix, M. (2010). Continuous use of authoring for adaptive educational hypermedia: A long-term case study. Paper presented at the 10th IEEE International Conference on Advanced Learning Technologies, ICALT 2010.Retrieved from http://dl.acm.org/citation.cfm?id=1846121 Gkatzidou, S., & Pearson, E. (2009). The potential for adaptable accessible learning objects: A case study in accessible vodcasting. Australasian Journal of Educational Technology, 25(2), 292-307. Grant, S., Aitchison, T., Henderson, E., Christie, J., Zare, S., McMurray, J., & Dargie, H. (1999). A comparison of the reproducibility and the sensitivity to change of visual analogue scales, Borg scales, and Likert scales in normal subjects during submaximal exercise. CHEST, 116(5), 1208-1217. Gu, X., Zha, C., Li, S., & Laffey, J. M. (2011). Design, sharing and co-construction of learning resources: A case of lifelong learning communities in Shanghai. Australasian Journal of Educational Technology, 27(2), 204-220. Hanson, W. E., Creswell, J. D., Creswell, J. D., Plano Clark, V. L., & Petska, K. S. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52(3), 224- 235. Karampiperis, P., & Sampson, D. (2006). Adaptive learning objects sequencing for competence-based learning. Paper presented at the 6th IEEE International Conference on Advanced Learning Technologies, Kerkrade, The Nederlands. Retrieved from http://dspace.learningnetworks.org/bitstream/1820/682/1/310karsam.pdf Kay, J. (2000). User modeling for adaptation. In C. Stephanidis (Ed.), User Interfaces for all (pp. 271- 294). Mahwah, NJ: Lawrence Erlbaum Associates. Kayama, M., & Okamoto, T. (2001). A Knowledge based navigation system with a semantic map approach. Educational Technology & Society, 4(2), 96-103. Klemke, R., Ternier, S., Kalz, M., & Specht, M. (2010). Implementing infrastructures for managing learning objects. British Journal of Educational Technology, 41(6), 873–882. Australasian Journal of Educational Technology, 2014, 30(1). 59 Kobsa, A., Koenemann, J., & Pohl, W. (2001). Personalised hypermedia presentation techniques for improving online customer relationships. The Knowledge Engineering Review, 16(2), 111-155. Lee, A. S. (1991). Integrating positivist and interpretive approaches to organizational research. Organization Science, 2(4), 342-365. Maier, P., Armstrong, R., Hall, W., & Ng, M. H. (2005). JointZone: users’ views of an adaptive online learning resource for rheumatology. Learning, Media and Technology, 30(3), 281-297. Maier, R., & Thalmann, S. (2007). Describing learning objects for situation- oriented knowledge management applications. Paper presented at the 4th Conference of Professional Knowledge Management (WM07), Potsdam. Mingers, J. (2000). Multimethodology: The theory and practice of combining management science methodologies.West Sussex, England: John Wiley & Sons. Mueller, D., & Strohmeier, S. (2011). Design characteristics of virtual learning environments: State of research. Computers & Education, 57(4), 2505–2516. O’Neil, A. F. (2008). The current status of instructional design theories in relation to today’s authoring systems. British Journal of Educational Technology, 39(2), 251–267. Ochoa, X., Cardinaels, K., Meire, M., & Duval, E. (2005). Frameworks for the automatic indexation of learning management systems content into learning object repositories. Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications, Montreal, Canada. Retrieved from http://ariadne.cti.espol.edu.ec/xavier/papers/Ochoa-EDMedia2005.pdf Oppermann, R. (1994). Adaptive user support - Ergonomic design of manually and automatically adaptable software (Vol. 40). Hillsdale, NJ: Lawrence Erlbaum Associates. Osborne, J. W., & Overbay, A. (2004). The power of outliers (and why researchers should always check for them). Practical assessment Research 9(6), 1-8. Paramythis, A., & Stephanidis, C. (2005). A generic adaptation framework for web-based hypermedia systems. In S. Y. Chen & G. D. Magoulas (Eds.), Adaptable and Adaptive Hypermedia Systems (pp. 80-102). Hershey: IRM Press. Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: Sage. Pawlowski, J. M., & Bick, M. (2006). Managing & re-using didactical expertise: The didactical object model. Educational Technology & Society, 9(1), 84-96. Pazzani, M. J., & Billsus, D. (2007). Content-based recommendation systems. In P. Brusilovsky, A. Kobsa & W. Nejdl (Eds.), The Adaptive Web (pp. 325–341). Berlin/Heidelberg: Springer-LNCS4321. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. Schoonenboom, J. (2012). Four scenarios for determining the size and reusability of learning objects. Australasian Journal of Educational Technology, 28(2), 249-265. Silverman, D. (1993). Interpreting qualitative data: Methods for analyzing talk, text, and interaction. London: Sage. Teddlie, C., & Tashakkori, A. (2009). Foundations of the mixed method research. Thousand Oaks, CA: Sage. Australasian Journal of Educational Technology, 2014, 30(1). 60 Triantafillou, E., Georgiadou, E., & Economides, A. A. (2007). Applying adaptive variables in computerised adaptive testing. Australasian Journal of Educational Technology, 23(3), 350-370. Triantafillou, E., Pomportsis, A., Demetriadis, S., & Georgiadou, E. (2004). The value of adaptivity based on cognitive style: An empirical study. British Journal of Educational Technology, 35(1), 95–106. Wee, H. L., Fong, K. Y., Tse, C., Machin, D., Cheung, Y. B., Luo, N., & Thumboo, J. (2008). Optimizing the design of visual analogue scales for assessing quality of life: A semi-qualitative study among Chinese-speaking Singaporeans. Journal of Evaluation in Clinical Practice, 14(1), 121-126. Yalcinalp, S., & Gulbahar, Y. (2010). Ontology and taxonomy design and development for Personalised web-based learning systems. British Journal of Educational Technology, 41(6), 883–896. Zimmermann, A., Specht, M., & Lorenz, A. (2005 ). Personalization and context management. User Modeling and User-Adapted Interaction, 15, 275-302. Zouaq, A., Nkambou, R., & Frasson, C. (2007). An integrated approach for automatic aggregation of learning knowledge objects. Interdisciplinary Journal of Knowledge and Learning Objects, 3(1), 135- 162. Corresponding author: Stefan Thalmann, Stefan.thalmann@uibk.ac.at Australasian Journal of Educational Technology © 2014. Please cite as: Thalmann, S. (2014). Adaptation criteria for the personalised delivery of learning materials: A multi-stage empirical investigation. Australasian Journal of Educational Technology, 30(1), 45-60.