Low domain knowledge readers navigating a hypertext: Shedding light on disorientation Australasian Journal of Educational Technology, 2020, 36(4). 189 Novices’ performance using hypertext materials: Shedding light on disorientation Álvaro Jáñez, Javier Rosales University of Salamanca Many researchers have proposed a causal link between low domain knowledge and disorientation in hypertext. Our objective was to challenge this hypothesis, examining other variables that might have an influence, such as design, reading instructions, or working memory capacity. We analysed navigation patterns and comprehension scores in a sample of low topic knowledge undergraduate students (N = 45) who were assigned to one of three hypertext conditions: reading for a test, reading to write a summary, and elaborative interrogation. Another sample (N = 45) of low knowledgeable undergraduate students were used as a control group, performing the same tasks using printed texts. Regarding disorientation, and contrasting with previous research, only a minority of students became disoriented in hypertext. Neither reading instructions nor working memory capacity had the expected impact on readers’ behaviour or outcomes, so hypertext design might be a key aspect on explaining disorientation. Implications for hypertext design and education are discussed. Implications for practice or policy: • The general assumption of low prior knowledge readers becoming disoriented in hypertext is falsified. • Educational hypertext designers should always consider hypertext structure, graphical overviews, and usability elements. • Educators must be aware that novices may show difficulties to adapt their learning behaviour to different reading goals. Keywords: hypertext; web navigation; disorientation; reading instructions; reading comprehension Introduction There is a growing body of research on the cognitive processes and design issues involved in hypertext and hypermedia navigation and comprehension (e.g., Amadieu et al., 2015; Burin, Barreyro, Saux, & Irrazábal, 2015; Chen, Fan, & Macredie, 2006; Jonassen, 1991; Rouet, 2006; Shapiro & Niederhauser, 2004; Stanton, 1994, Suchman, 1987). Many variables have been identified as affecting reading performance in hypertext, but we do not have all the answers yet and researchers are still reporting difficulties in ensuring efficient navigation with no disorientation problems. For example, Li, Tseng, and Chen (2016) tried to use the beneficial aspects of highlighting to improve hypertext comprehension and navigation, but they found the opposite, reporting navigation difficulties of participants. Another study by Meyers and Bagnall (2015) found a unique multidimensional disorientation, and concluded further research was needed to identify mitigating factors for multidimensional disorientation. This research takes a step forward on the disorientation research by analysing and expanding knowledge of how novices perform using different materials under different instructions. Several authors suggest that low prior knowledge causes disorientation in hypertext/hypermedia environments (Last, O'Donnell, & Kelly, 2001; Lawless & Kulikowich, 1996; Rezende & de Souza Barros, 2008). However, potential solutions to facilitate novices’ navigation are scarce. In an educational setting where the Internet is a primary source of information, it is necessary to investigate these issues. In this study, hypertext design elements were manipulated in order to reduce or eliminate disorientation in undergraduates studying science topics. Only novices were considered for the experiment and they were tested using different reading tasks; analysing the potential benefits of one reading instruction over the others on learning outcomes. The impact of different levels of working memory capacity on novices’ performance was also tested. Australasian Journal of Educational Technology, 2020, 36(4). 190 Hypertext navigation and disorientation problems Disorientation has been considered a major problem of hypertext use for decades (Conklin, 1987; Shih, Huang, Hsu, & Chen, 2012; Thüring, Hannemann, & Haake, 1995). Some common examples of this problem are: not knowing where you are in the network, not knowing where to go next, or having trouble getting to where you want to go. The main reasons disorientation problems are so important are because they have a negative impact on navigation efficiency (McDonald & Stevenson, 1996) and they hinder learning performance (Amadieu, Tricot, & Mariné, 2010). Navigation performance, and disorientation, have been previously investigated through the analysis of navigation profiles (e.g., Rezende & de Souza Barros, 2008). A navigation profile shows every node that has been visited by a reader, the time spent on each node, and the order in which nodes were accessed during a reading session. Navigation profiles are unique for each person and for each reading session, therefore, several authors have used cluster analysis to extract navigation patterns in which any reader's navigation profile can be classified. For example, Lawless and Kulikowich (1996) identified three navigation patterns: knowledge seekers, feature explorers, and apathetic hypertext users. The knowledge seekers group had the highest domain knowledge and showed very sophisticated navigation profiles. The feature explorers spent most of their time exploring the hypertext and its functioning. The apathetic hypertext users had the lowest domain knowledge scores and spent short intervals of time interacting with the text, showing no interest in gathering information or exploring the hypertext features. Similar patterns were found by Barab, Bowdish, and Lawless (1997) and Rezende and de Souza Barros (2008). Research on navigation patterns suggests that prior domain knowledge is an important factor influencing the navigation paths taken by hypertext readers, and that some topic knowledge is necessary to interact adequately with a hypertext because students who are topic novices are bound to become disoriented while navigating. Regarding prior knowledge, this research expected to obtain more encouraging results: that is that being a novice should not be synonym to becoming disoriented in hypertext. Other variables, such as hypertext design or reading instructions, may help explain the negative impact that being a novice is supposed to have. Also, working memory capacity is an important factor. Making decisions about which links to follow and the non-linear reading experience are specific factors that may overwhelm working memory capacity during hypertext navigation (DeStefano & LeFevre, 2007). In fact, readers with low working memory capacity perform better using traditional linear text than hypertext (Lee & Tedder, 2003). Effects of working memory capacity can also impact hypertext design, if it fosters relational processes appropriately (Amadieu, Lemarié, & Tricot, 2017). Previous studies have not investigated these variables in combination. Hypertext design elements that facilitate novices’ performance It is widely accepted that knowledgeable students are better able to navigate easily, experiencing less disorientation problems and reporting more positive feelings about using non-linear materials (Last et al., 2001). But, how can navigation be facilitated to low domain knowledge readers? Several researchers have found some design features (such as hypertext structure, graphical overviews, and Web usability) that can help achieve this goal. For example, some studies indicate that participant navigation is more efficient when using a hierarchical hypertext than when using a non-linear hypertext (Burin et al., 2015). Hierarchical hypertexts also increase navigation efficiency when compared to linear or randomly structured hypertexts (Schoon & Cafolla, 2002). When prior knowledge is controlled, readers with low prior knowledge obtain better comprehension scores when using a hierarchical hypertext rather than a linear hypertext (Calisir & Gurel, 2003) or a mixed structured hypertext (Calisir, Eryazici, & Lehto, 2008). Since structuring the content hierarchically seems to be beneficial for novices, this research implemented this structure in the experimental materials. Graphical overviews act as a map of the hypertext system, reflecting conceptual relationships and/or connections among nodes. Previous findings suggest that the mere presence of a graphical overview is not enough to facilitate navigation or improve learning (Müller-Kalthoff & Möller, 2003; Vörös, Rouet, & Pléh, 2011), therefore researchers have focused on finding out which graphical overviews are better suited to this end. Potelle and Rouet (2003) found that low prior knowledge readers obtained better comprehension with a hierarchical structured graphical overview than with a networked or alphabetic Australasian Journal of Educational Technology, 2020, 36(4). 191 structure. Amadieu, van Gog, Paas, Tricot, & Mariné (2009) obtained similar results: low prior knowledge participants obtained more conceptual knowledge and showed less mental effort and less disorientation with the hierarchical overview than with the network overview (see Amadieu and Salmerón (2014), for a review on graphical overviews), therefore this research implemented this type of graphical overview in the experimental materials. Finally, some aspects of Web usability are considered to reduce disorientation. Following Nielsen (2000; 2004; 2008), the experimental materials used in this research provided a list of links in each node, rather than inserted links in the text. The links list were left justified to facilitate scanning and reading, and were different colours for visited (purple) and unvisited (blue) links. The links also had the same names as the respective section title and changed to a bright red and with underlining when the mouse was passed over them, enabling the user to easily discern the chosen link. Effects of reading instructions Research on the effects of reading instructions (using printed texts) has a long tradition in psychology (see McCrudden and Schraw (2007) for a revision), but research on this topic is scarce in hypertext. Also, prior knowledge is not usually manipulated in these experiments. Because this variable may offer some insights on why novices do not perform as expected, some studies were reviewed investigating the topic on traditional texts, focusing on the instructions manipulated in the experiment: elaborative interrogation, reading for a test, and reading to write a summary. In elaborative interrogation research, "Why...?" questions are inserted in the text. These questions are supposed to increase readers' attention, active processing of information, and comprehension obtained during the elaboration process (Levin, 2008). Previous studies have found that the elaborative interrogation technique improves comprehension when compared to reading the text without the elaborative interrogation questions (Smith, Holliday, & Austin, 2010), especially for low prior knowledge readers (Ozgungor & Guthrie, 2004). However, other studies show negative effects of elaborative interrogation compared to reading the text twice (Clinton, Alibali, & Nathan, 2016). Van den Broek, Lorch, Linderholm, & Gustafson (2001) investigated the process of expository text reading through think-aloud protocols. They manipulated dual purpose instructions: reading for a test, and reading for pleasure. Participants with a study purpose recalled 20% more information than participants reading for pleasure. Finally, Bråten and Samuelstuen (2004) manipulated three reading purposes: reading for a test, reading to write a summary, and reading to discuss text content with peers. They found that the more the prior knowledge, the more reported used of strategies, suggesting that low prior knowledge may impede or limit the appropriate application of reading strategies - a hypothesis that was tested in this research. Other researchers (Hagen, Braasch, & Bråten, 2014) have found that a summary goal hindered performance compared to a constructing an argument goal. As previously mentioned, the effects of different reading instructions have been barely investigated in the context of hypertext, consequently there is not an established methodology about how to assess reading strategy use in hypertext reading. However, several authors have used implicit measures of user behaviour to predict user preferences given a particular reading task (Fox, Karnawat, Mydland, Dumais, & White, 2005). The low performance of novices with hypertext may be related to the type of reading instruction they are given. Therefore, several instructions within the experimental material were manipulated to examine the potential effects each has on novices’ performance. Hypotheses Testing whether or not novices’ performance is hindered using hypertext compared to traditional printed linear texts. As reviewed in previous sections, low prior knowledge is considered an essential factor in causing disorientation in hypertext (Last et al., 2001; Lawless & Kulikowich, 1996; Rezende & de Souza Barros, 2008) in turn, hindering performance. Previous findings regarding hypertext design were applied to construct a hypertext which can be navigated with no problems by most low prior knowledge readers. In Australasian Journal of Educational Technology, 2020, 36(4). 192 this research, a control group performed the same tasks using printed materials in the usual linear structure which is considered to pose no negative impacts on learners. It was hypothesized that experimental groups would obtain similar comprehension scores to control participants and their hypertext navigation patterns would not show disorientation issues. If both criteria were met, results would falsify the general assumption of low prior knowledge causing disorientation in hypertext. Testing whether or not novices’ performance varies when dealing with different tasks in hypertext compared to traditional texts Three reading instructions were used: reading for a test, reading to write a summary, and elaborative interrogation. According to previous findings, it was hypothesized that participants under elaborative interrogation and test instructions would obtain better comprehension scores than participants under summary instructions, and participants under summary instructions would focus on nodes that were higher in the hypertext structure, since they offered more general information. Also, participants reading for a test were expected to show more monitoring strategies (such as making more revisits to nodes). Therefore, the hypothesis was that different navigation behaviours would be found for each reading instruction. Reading comprehension outcomes were expected to be similar to participants in the control group, but different across instructions. Testing whether or not working memory capacity has an impact on novices’ navigation behaviour As reviewed, it was hypothesized that if materials are well designed, working memory capacity would not impose any problems on readers’ behaviour. However, if materials are not well designed, higher working memory capacity participants would have lower disorientation problems. To test these hypotheses an online measure (navigation paths) and an offline parameter (comprehension scores) were used in order to assess disorientation, navigation strategies, and performance. Method Participants Participants were 90 undergraduate students (86.7% female, 13.3% male) from the Faculty of Education at a Spanish university (mean age 21.6 years). Half of the participants formed the control groups, and the other half the experimental groups. All participants had low prior knowledge about the text topic, as indicated by a prior knowledge test. Participants were equally distributed in the three groups according to their working memory span scores. High, medium, and low levels of working memory capacity were assessed by assuming the 33.3 and 66.6 percentiles of the scores distribution as cut-off points. Members of each of the groups were randomly assigned to one of six conditions resulting from a 2 x 3 design with text format (hypertext vs. printed text) and reading instructions (elaborative interrogation, test taking, summary writing) as between-subject factors. This design ensured all conditions had the same proportion of high, medium and low working memory capacity participants. Ethics guidelines from the institution were met, by obtaining informed consent from all participants prior to the experiments and anonymising all data. Materials Reading span test Each participant was assessed individually for working memory capacity using an adapted to Spanish version of the Reading Span Test (Daneman & Carpenter, 1980; Elosúa, Gutiérrez, García Madruga, Luque, & Garate, 1996). Sequences of short sentences with no semantic or structural relations were presented one at a time. During the task, the participant was required to read aloud each sentence and remember the last word of each of them. No time elapsed between consecutive sentences, so the participants were not able to use mental repetition of the target words. The test started with a sequence of two sentences, and the number of sentences was increased during the test to a maximum of six. The test started with three practice sequences of two sentences each to make sure participants understood the task, and ended when the participant failed to recall the three sequences of a same level. A bonus was given if the words were recalled in the order of presentation. Australasian Journal of Educational Technology, 2020, 36(4). 193 A PowerPoint was used to deliver this test: white background and black font (size: 12 point); sentences appearing in the middle of the screen, just one line in length (13-16 words). The researcher conducting the experiment passed the slides forward the moment the participant finished reading each sentence. Instructions were written at the beginning of the presentation. The same researcher assessed all participants. Hypertext and printed text An expository hypertext was constructed in Spanish as a webpage (coded in HTML and using CSS), dealing with the topic of rare illnesses. The hypertext was approximately 2,500 words in 21 nodes (plus a node with the reading instructions and another node with the graphical overview). As previously mentioned, the hypertext was designed according to what research shows beneficial for low domain knowledge students. The hierarchical structure was used, as it benefits low prior knowledge readers navigation (Burin et al., 2015). Figure 1 shows the specific structure of the hypertext. Lines in Figure 1 represent connections between nodes. As we can see, every node gives access to its parent, child, and sibling nodes. Figure 1. Graphical overview depicting the structure of the hypertext A hierarchical graphical overview was included (Figure 1) to facilitate navigation to the target readers (see Amadieu & Salmerón, 2014). This graphical overview was non-navigable (in order to force readers to navigate through text nodes, using the hyperlinks) and was presented as an individual node in the network. Two buttons were added at the end of every node to give direct access to this overview, as well as to the instructions page. Australasian Journal of Educational Technology, 2020, 36(4). 194 Finally, some aspects of web usability were also considered, such as having different colours for visited and unvisited links, left-justify the links list, or avoiding to use the same starting words (Nielsen, 2000; 2004; 2008). A screen capture of the layout can be seen in Figure 2. The browser used was Mozilla Firefox, and all functions were accessible (markers, option of multiple tabs/windows, etc.) and participants were explicitly told they could use them, to navigate as they usually do (however, only the backward and forward functions of the browser were used). Figure 2. Screen capture of the Web page layout, showing the basic structure of each node (title, text, links list, and buttons). Unvisited links are blue in colour and visited links are purple. Printed texts were the same as the hypertext (same contents, font size, font type, and spacing), including all headings and the same graphical overview. The only difference was that printed texts were written in a linear format, while in the hypertext there were hyperlinks to navigate through the different sections. The printed text starts with the instructions page, after that the graphical overview (as in the hypertext) and then all sections one after another following a linear order. The graphical overview in Figure 1 allows a better understanding of this order of sections: Rare Illnesses, Lyme Disease, Information on Borrelia, Method of transmission of Lyme disease, Lyme disease in humans). The final product was a document of 10 stapled pages. All conditions (elaborative interrogation, exam, summary) used the same text/hypertext, but the elaborative interrogation group had "Why ...?" questions at the end of each section/node. Prior knowledge and comprehension tests A test was created to assess prior domain knowledge about the text topic. Five short answer questions about the main key concepts were included. For example, participants were asked to define “nematode” or “espiroquete”, which appeared (but were not explained) in the text, or to name similar illnesses to the ones that appeared on the text. To ensure that all participants had low prior knowledge, answering two or more questions correctly caused the exclusion of that participant from the sample. Eight participants were excluded this way, and they were substituted by similar participants (on working memory capacity) from the same Faculty. Further exclusions were not necessary. This test was performed after reading the text/hypertext materials, since pre-testing might cause students to assess the pre-tested contents as those that are most important, affecting their reading behaviour (McCrudden, Schraw, & Kambe, 2005). Another test was created to assess comprehension of the material after the reading task. Twelve short- answer questions evaluated text-based comprehension by requesting information that was stated in a single node/section. An example of this type of question is the following: "What animal transmits Lyme disease to humans?". Seven essay questions evaluated situation model comprehension by requesting linking information about at least two nodes/sections with prior knowledge. An example of this type of question is: "Analyse your lifestyle and explain which of the illnesses mentioned in the text you could contract more easily, and which one you need not to worry about. Justify your answer." An assessment Australasian Journal of Educational Technology, 2020, 36(4). 195 rubric was design and all answers were rated by an expert on the topic. All participants completed the same test regardless of their reading instructions, and all groups (experimental groups reading the hypertext and control groups reading the printed text) answered the test in paper format. Reading instructions Participants in both groups (control participants using printed texts and experimental participants using hypertext) were assigned to one of three reading instructions: elaborative interrogation, test taking, and summary writing. The instructions were presented on the first page/node. Participants under the elaborative interrogation condition were instructed to read the text/hypertext freely, their only premise was to answer every question they found during the reading session. Participants under test taking instructions were asked to learn as much as possible from the text, since they would be tested at the end. Finally, participants under summary instructions were asked to read the text with the purpose of writing a summary of the main ideas after the reading session. Data analysis Screen recording software was used to capture every action performed by participants during the reading session. Data were collected from these recordings, such as reading order and time spent in each node visit. This information was plotted into graphs as a function of time, as it reflects the structure of the navigation. Cluster analysis was performed on navigation profiles in order to extract navigation patterns. Visual inspection of navigation profiles (graphs and data) was used to analyse navigation similarities and differences between the clusters identified, looking for signs of disorientation: number of missed (not visited) nodes, incoherent transitions among nodes and/or many short (less than 5 seconds) visits to the same nodes. Missing relevant nodes is a sign of disorientation. Incoherent transitions among nodes can be interpreted as the reader not knowing where to go next, hence making an incoherent transition that will affect comprehension. Many short visits to the same node are also a common sign of disorientation. Short visits to referential points (such as the graphical overview) are not considered disorientation, but repeatedly visiting (for less than 5 seconds) a dead-end node, from where no other information can be accessed, suggests that the reader is looking for some piece of information and not finding the correct path. A chi-square test was conducted to check if navigation patterns had been affected by the instructions or by working memory capacity. After that, two ANOVAs were performed to check for interactions between navigation patterns and comprehension scores, and between text format (hypertext/printed text) and comprehension scores. Procedure First, working memory capacity was assessed individually in all participants using the Reading Span Test. The same number of high, medium, and low working memory capacity participants was used to form three experimental groups (elaborative interrogation, exam, and summary instructions reading the hypertext) and three control groups performing the same tasks with printed texts. All participants were given 30 minutes to read the text/hypertext, and right after completion of that time, they had to fulfil the comprehension test (with no time restrictions). Results Navigation patterns First of all, the navigation profiles of participants were tested for later analysis if they were impacted by other variables. Following Barab et al. (1997), we used Ward's hierarchical clustering technique to examine navigation profiles of experimental participants (hypertext users). The analysis revealed 5 different clusters and, after visual inspection of the navigation profiles assigned to each cluster, three global navigation patterns were identified. Linear navigation (N = 26) Participants in clusters 1, 4, and 5. Figures 3, 4, and 5, respectively, show examples of navigation profiles in these clusters. Figure 1 also provides a visual understanding of how the sections referred to in the y- axis were structured in the hypertext. All participants showed a similar navigation. Cluster analyses separated these groups probably due to their reading times. Participants in cluster 1 (Figure 3) took around 1,300 seconds on the first reading, participants in cluster 4 (Figure 4) took between 800 - 1,100 Australasian Journal of Educational Technology, 2020, 36(4). 196 seconds on the first reading, and participants in cluster 5 (Figure 5) spent most of their time on just one reading. Despite this fact, all participants in these clusters showed a linear navigation, reading all nodes and following a coherent order. The y-axis in the Figures show the hypertext nodes in a coherent order from bottom to top. An ascending line in the graph without skipping sections means the participant followed the most coherent order. The visible downfalls in Figures 3, 4, and 5, are normal, as they happen in dead-end nodes, where some backtracking is necessary to have access to new nodes. Figure 3. Sample navigation profile of a participant in cluster 1 (linear navigation) Figure 4. Sample navigation profile of a participant in cluster 4 (linear navigation) Australasian Journal of Educational Technology, 2020, 36(4). 197 Figure 5. Sample navigation profile of a participant in cluster 5 (linear navigation) Linear with minor disoriented navigation (N = 13) Participants in cluster 2 (Figure 6) showed a linear, coherent navigation, but skipping some nodes on the first reading. For example, Lyme disease in humans and its child nodes, and Information on toxocara and its child nodes are skipped at around the 400 seconds. These jumps across the material are considered minor disorientation rather than non-linear navigation because they happen mostly in dead-end nodes. For example, the participant in Figure 6, the last section visited before the first jump is Method of transmission, which requires backtracking to access the next coherent node: Lyme disease in humans. Instead of doing so, the participant forgot there were more sections to read about Lyme disease, he/she was disoriented, so used the direct access to the graphical overview as a shortcut, since it was a visit of less than 2 seconds, to start reading the next illness. Also, this is considered minor disorientation rather than heavy disorientation because these participants, after finishing reading the last illness in the text, realise they skipped some sections, and find their way without any problem to those unread nodes. All skipped sections are visited between seconds 1,000 to 1,400 without hesitation on how to get there, and those subsections were read in a linear coherent order. Australasian Journal of Educational Technology, 2020, 36(4). 198 Figure 6. Sample navigation profile of a participant in cluster 2 (linear with minor disoriented navigation) Disoriented navigation (N = 6) Participants in cluster 3 (Figure 7) showed a disoriented navigation. These participants did not show linearity in their reading, chose incoherent paths (e.g., reading treatments before knowing the symptoms), making many short visits to nodes, and leaving some nodes unvisited. The participant in Figure 7 left unread all of "Toxocariasis in humans" subsection, which included epidemiology, risk factors, symptoms and treatment of toxocariasis. Not all participants with a disoriented navigation leave unread sections, but all of them share non-linearity, incoherent paths and many short visits and jumps across the material. Figure 7. Sample navigation profile of a participant in cluster 3 (Disoriented navigation) Australasian Journal of Educational Technology, 2020, 36(4). 199 Most participants showed a linear, coherent navigation with no, or minor, disorientation problems. The data was then examined for any potential effects of other variables on navigation patterns. Considering that the type of instruction used (elaborative interrogation, exam, or summary) was a categorical variable and that the navigation profiles identified was also categorical (linear, linear with minor disorientation, or disoriented), a chi-square test for independence was performed. Relevance instructions did not affect navigation patterns, as similar proportions of linear, linear with minor disorientation, and disoriented patterns were found in each condition (X2 (4, N = 45) = 1.615, p > .05). Therefore, the specific paths taken during reading (coherence of reading orders) or the amount of time devoted to specific nodes were not affected by reading instructions. Similarly, considering working memory capacity as a categorical variable (low, medium and high), another chi-square test for independence was performed. WM had also no effects on navigation patterns (X2 (4, N = 45) = 5.154, p > .05). Navigation behaviour As we have seen, reading instructions had no effect on navigation patterns, and this was a good first indicator that the navigation strategies used by readers were similar across all reading instructions. However, previous findings agree that different reading instructions are related to different reading behaviours, so further analyses were performed to check for potential differences among participants under different reading instructions. An ANOVA showed no significant differences on total transitions made (F (2,42) = 1.845, p > .05; this variable needed a logarithmic transformation to meet the normality assumption), so review patterns (revisits of the material) were shown to be similar across instructions. No significant differences were found either on total time devoted to the graphical overview (F (2,42) = 0.620, p > .05; this variable needed a logarithmic transformation to meet the normality assumption), so this referential point was similarly used across participants despite some of them had a “write a summary” or “study” instruction that could have triggered a higher use of this node. Finally, no differences were found on percentage of visits to higher nodes in the hypertext hierarchical structure (F (2,42) = 0.769, p > .05), which had been predicted to be influenced by reading instructions; that is the “write a summary” instruction group should have focused on these higher nodes were the general information of the text can be obtained. Comprehension scores After analysing reading behaviours, readers’ performance was analysed (Table 1). After checking that the assumptions of normality and homogeneity of variances were met, ANOVAs were conducted to see if there would be an interaction between text format and reading instructions for comprehension scores. One rater scored all tests, and a second rater scored a third of the tests extracted randomly from the whole sample. Inter-rater reliability was high (r = .86). No significant effects were found of reading instructions on hypertext comprehension at text base (F (2,42) = 0.360, p > .05) or situation model levels (F (2,42) = 0.890, p > .05), or on printed text comprehension at text base (F (2,42) = 0.159, p > .05), or situation model (F (2,42) = 0.771, p > .05). Table 1 Mean comprehension scores (0-100) of all groups Reading instruction Comprehension level Group Elaborative interrogation Exam Summary Text base Hypertext 61.38 61.85 58.28 Printed text 60.94 58,87 62.07 Situation model Hypertext 48.53 39.92 40.45 Printed text 43.92 44.25 38.25 As reading instructions had no effects on navigation or comprehension, and for clarity reasons, we will refer to hypertext group and printed group in the rest of the analysis. As indicated in the hypotheses, the data was next analysed to check if there were differences in performance between the hypertext group and the printed text group. No significant differences between text format were found on text base (F (1,88) = 0.002, p > .05) or situation model (F (1,88) = 0.009, p > .05) comprehension scores. Finally, comprehension scores were analysed only in the hypertext group to Australasian Journal of Educational Technology, 2020, 36(4). 200 check if navigation patterns had effects on them. Figure 8 shows mean comprehension scores for all navigation patterns. Figure 8. Mean hypertext comprehension scores for each navigation pattern The ANOVA showed no significant differences among the three navigation groups at text-base comprehension level, F (2,42) = 0.576, p > .05, but differences were significant at the situation model comprehension level, F (2, 42) = 5.684, p = .007. Tukey post hoc comparisons indicated that participants with a linear navigation had better comprehension scores than participants with a linear with minor disoriented navigation (p = .038) and with disoriented navigation (p = .022) patterns. Participants in the linear with minor disoriented navigation had better situation model comprehension scores than participants with disoriented navigation, but differences were not statistically significant (p > .05). Discussion Testing whether or not novices’ performance is hindered using hypertext compared to traditional printed linear texts The first hypothesis challenged the assumption of low prior knowledge causing disorientation problems in hypertext. Contrasting with previous research (Burin et al., 2015; Last et al., 2001; Lawless & Kulikowich, 1996; Rezende & de Souza Barros, 2008) this hypothesis was partially correct. The majority of our sample did not show the slightest sign of disorientation. These participants (classified as linear navigators) followed a coherent reading order, read all hypertext nodes, and showed no random transitions. Also, participants in the hypertext condition obtained similar comprehension outcomes than controls using traditional linear texts. Therefore, both online (reading behaviour) and offline (comprehension scores) measures suggest that most participants were not disoriented using hypertexts. This result falsifies the general assumption about low prior knowledge causing disorientation. A simple and clear hypertext design seems to eliminate disorientation in a majority of low prior knowledge readers, irrespective of their gender, working memory capacity, or reading instructions. Therefore, the results show that novices using hypertext can perform equally well, in general, compared to when using printed materials. Despite this success, a minority of the participants (13%) still showed heavy disorientation problems. These participants followed incoherent reading orders, random transitions among nodes, and their comprehension was hindered compared to linear navigators or to control participants reading the printed Australasian Journal of Educational Technology, 2020, 36(4). 201 texts. This suggests that hypertext navigation (and, therefore, disorientation), as many other complex activities, cannot be explained by one factor alone. The role of new variables must be explored in combination with prior knowledge, working memory capacity, and hypertext design to better understand navigation performance. Also, time exposure to hypertext materials and the extension of such materials should be further investigated, since longer duration of reading and more extensive texts may have different effects on performance, and may have different effects depending on students' prior knowledge. Testing whether or not novices’ performance varies when dealing with different tasks in hypertext compared to traditional texts The second hypothesis analysed the navigation strategies used by low prior knowledge students while reading a hypertext under different instructions. The effects of reading instructions have been systematically found by many researchers (see McCrudden & Schraw, 2007). Reading instructions affect the reading strategies used and readers' performance. However, no effects of reading instructions on navigation or comprehension scores were found. Hypertext format would appear to have nothing to do with this, since the control group using printed texts showed also no differences in performance across reading instructions. This lack of effect contrasts with previous research, especially in the case of elaborative interrogation, which is supposed to have significant beneficial effects for low knowledgeable readers (Ozgungor & Guthrie, 2004). One explanation is that research on elaborative interrogation usually compares this technique against reading the text twice (Ozgungor & Guthrie, 2004; Smith et al., 2010). In this study, the elaborative interrogation technique was compared with more demanding tasks, such as reading for a test, and reading to write a summary. Therefore, the benefits of elaborative interrogation might only appear when compared to low demanding tasks, or when it is used with less structured materials. This lack of effect of elaborative interrogation has been reported by other researchers using both printed texts (Callender & McDnaiel, 2007; Clinton et al., 2016) and electronic linear texts (Dornisch & Sperling, 2006). The effects of elaborative interrogation should be further explored in the future, taking into account prior knowledge, hypertext structure, and a wider range of reading instructions to fully understand how it works and when it is most useful. It was also interesting to note the lack of effect of the test taking condition. This group was the only one that knew that there would be an evaluation after the reading session, and expecting to be evaluated has been found to increase recall (van den Broek et al., 2001). But no effects of this instruction on navigation performance or comprehension scores were found. Regarding the instruction of reading to write a summary, lack of effect was even more interesting. Participants under summary writing instructions tend to use more organisational strategies (Bråten & Samuelstuen, 2004), but the participants in this research used the graphical overview as much as participants in the other groups. They did not make more transitions and they did not focus on higher nodes in the hierarchy, where the information offered is more general. All this data suggests that having low prior knowledge about the topic, and not the digital or traditional format of the materials, affects strategy implementation. These results support the findings by Jáñez and Rosales (2016) and by Potelle and Rouet (2003) who suggest that low knowledgeable participants may need to read and explore all the materials in the most coherent order possible, regardless of their reading instructions, presumably to create a mental representation of all the information available and how it is structured before attempting, or being capable, of implementing strategies more directly related to achieving the task objectives. Of course, other interpretations are also valid, so future research using different methodologies, such as think aloud protocols, is necessary to confirm/falsify and extend these results. Testing whether or not working memory capacity has an impact on readers’ navigation behaviour Working memory capacity showed no impact on readers’ behaviour, contrasting with previous findings (DeStefano & LeFevre, 2007; Lee & Tedder, 2003). As suggested by Amadieu et al. (2017), a potential explanation is that the materials were properly designed, so working memory capacity was not overwhelmed during the task. It would be interesting to test different aspects of working memory other Australasian Journal of Educational Technology, 2020, 36(4). 202 than reading span, such as spatial WM (Kornmann et al, 2016) or working memory updating (Hahnel, Goldhammer, Kröhne, & Naumann, 2017) to find new answers. Conclusion This study aimed to analyse novices’ performance in hypertext. The results reported here suggest the great importance of implementing specific hypertext features to facilitate navigation of a target population. From the results, it seems that getting lost might be due to poor design, not from the nature of hypertext. Teachers and designers of educational materials should keep in mind that hypertext structure, graphical overviews, and usability are of key importance when designing digital materials, since novices seem to benefit from very specific design features. For example, we showed how participants classified as linear with minor disorientated navigation, got lost almost exclusively when reaching dead-end nodes, so some navigation support should be implemented in those sections, or directly avoid them. Regarding strategy implementation, we assessed some of the navigation strategies used by hypertext readers through implicit measures. Our results suggest that novices are not able to implement efficient strategies to achieve goals, since all participants used very similar navigation strategies and performed similarly despite having very different reading instructions. We do not have data on reading behaviour of readers using printed materials, but considering that their outcomes were similar to readers using hypertext, there is a high possibility that their behaviour is also similar. Therefore, teachers should focus on offering a global view of the contents and key concepts of the materials used before asking students to perform any task. Otherwise, students might not be able to implement efficient strategies for the task at hand. Students, on the other hand, are relying more and more on online educational materials. They might benefit from having some guidelines on which materials to use: those that include a graphical overview depicting the contents, which offer clear navigational cues on which links have been visited and which links have not and that are structured in a way they can understand. Considering the international assessments that are periodically performed, these issues are of great importance: we need to be sure that we are assessing performance and that variables such as disorientation are not impacting students’ results, affecting the interpretation we make of the data. To that end, we need to explore these matters further to expand our knowledge on the topic, trying to control several variables in comprehensive experiments, such as prior knowledge, working memory capacity, hypertext design, and task objectives. Also, navigation performance should be analysed using a variety of online and offline measures, in order to fully understand the implications of hypertext users' behaviour. We believe that more investigation in this area will lead us to know how to use hypertexts more efficiently and how to unlock the potential benefits for learning that this tool is supposed to possess. Acknowledgements We would like to thank Jean-François Rouet for his thorough review and helpful comments on an early draft of this manuscript. References Amadieu, F., Lemarié, J., & Tricot, A. (2017). How may multimedia and hypertext documents support deep processing for learning? Psychologie française, 62(3), 209-221. https://doi.org/10.1016/j.psfr.2015.04.002 Amadieu, F. & Salmerón, L. (2014). Concept maps for comprehension and navigation of hypertexts. In R. Hanewald & D. Ifenthaler (Eds), Digital knowledge maps in education (pp. 41-59). New York, NY: Springer. Amadieu, F., Salmerón, L., Cegarra, J., Paubel, P.-V., Lemarié, J., & Chevalier, A. (2015). Learning from concept mapping and hypertext: An eye tracking study. Educational Technology & Society, 18(4), 100-112. Retrieved from https://www.learntechlib.org/p/175145/ https://doi.org/10.1016/j.psfr.2015.04.002 https://www.learntechlib.org/p/175145/ Australasian Journal of Educational Technology, 2020, 36(4). 203 Amadieu, F., Tricot, A., & Mariné, C. (2010). Interaction between prior knowledge and concept-map structure on hypertext comprehension, coherence of reading orders and disorientation. Interacting with Computers, 22(2), 88–97. https://doi.org/10.1016/j.intcom.2009.07.001 Amadieu, F., van Gog, T., Paas, F., Tricot, A., & Mariné, C. (2009). Effects of prior knowledge and concept-map structure on disorientation, cognitive load, and learning. Learning and Instruction, 19(5), 376-386. https://doi.org/10.1016/j.learninstruc.2009.02.005 Barab, S., Bowdish, B., & Lawless, K. (1997). Hypermedia navigation: Profiles of hypermedia users. Educational Technology Research & Development, 45(3), 23-41. https://doi.org/10.1007/BF02299727 Bråten, I., & Samuelstuen, M. S. (2004). Does the influence of reading purpose on reports of strategic text processing depend on students’ topic knowledge? Journal of Educational Psychology, 96(2), 324– 336. https://doi.org/10.1037/0022-0663.96.2.324 Burin, D. I., Barreyro, J. P., Saux, G., & Irrazábal, N. C. (2015). Navigation and comprehension of digital expository texts: Hypertext structure, previous domain knowledge and working memory capacity. Electronic Journal of Research in Educational Psychology, 13(3), 529-550. https://doi.org/10.14204/ejrep.37.14136 Calisir, F., Eryazici, M., & Lehto, M. R. (2008). The effects of text structure and prior knowledge of the learner on computer-based learning. Computers in Human Behavior, 24(2), 439-450. https://doi.org/10.1016/j.chb.2007.01.032 Calisir, F., & Gurel, Z. (2003). Influence of text structure and prior knowledge of the learner on reading comprehension, browsing and perceived control. Computers in Human Behavior, 19(2), 135-145. https://doi.org/10.1016/S0747-5632(02)00058-4 Callender, A. A., & McDaniel, M. A. (2007). The benefits of embedded question adjuncts for low and high structure builders. Journal of Educational Psychology, 99(2), 339-348. https://doi.org/10.1037/0022-0663.99.2.339 Chen, S. Y., Fan, J.-P., & Macredie, R. D. (2006). Navigation in hypermedia learning systems: Experts vs. novices. Computers in Human Behavior, 22(2), 251-266. https://doi.org/10.1016/j.chb.2004.06.004 Clinton, V., Alibali, M. W., & Nathan, M. J. (2016). Learning about posterior probability: Do diagrams and elaborative interrogation help? The Journal of Experimental Education, 84(3), 579-599. https://doi.org/10.1080/00220973.2015.1048847 Conklin, J. (1987). Hypertext: An introduction and survey. IEEE Computer, 20(9), 17-41. https://doi.org/10.1109/MC.1987.1663693 Daneman, M., & Carpenter, P. A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19(4), 450-466. https://doi.org/10.1016/S0022- 5371(80)90312-6 DeStefano, D., & LeFevre, J.-A. (2007). Cognitive load in hypertext reading: A review. Computers in Human Behavior, 23(3), 1616-1641. https://doi.org/10.1016/j.chb.2005.08.012 Dornisch, M. M., & Sperling, R. A. (2006). Facilitating learning from technology enhanced text: Effects of prompted elaborative interrogation. Journal of Educational Research, 99(3), 156-165. https://doi.org/10.3200/JOER.99.3.156-166 Elosúa, M. R., Gutiérrez, F., García Madruga, J. A., Luque, J. L., & Garate, M. (1996). Adaptación española del "Reading Span Test" de Daneman y Carpenter. Psicothema, 8(2), 383-395. Retrieved from http://www.psicothema.com/psicothema.asp?id=37 Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2), 147-168. https://doi.org/10.1145/1059981.1059982 Hagen, Å. M., Braasch, J. L. G., & Bråten, I. (2014). Relationships between spontaneous note-taking, self-reported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, 37(S1), S141-S147. https://doi.org/10.1111/j.1467- 9817.2012.01536.x Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2017). Reading digital text involves working memory updating based on task characteristics and reader behaviour. Learning and Individual Differences, 59, 149-157. https://doi.org/10.1016/j.lindif.2017.09.001 Jáñez, Á., & Rosales, J. (2016). Novices' need for exploration: Effects of goal specificity on hypertext navigation and comprehenison. Computers in Human Behavior, 60, 121-130. https://doi.org/10.1016/j.chb.2016.02.058 Jonassen, D. H. (1991). Hypertext as instructional design. Educational Technology Research and Development, 39(1), 83-92. https://doi.org/10.1007/BF02298109 https://doi.org/10.1016/j.intcom.2009.07.001 https://doi.org/10.1016/j.learninstruc.2009.02.005 https://doi.org/10.1007/BF02299727 https://psycnet.apa.org/doi/10.1037/0022-0663.96.2.324 http://dx.doi.org/10.14204/ejrep.37.14136 https://doi.org/10.1016/j.chb.2007.01.032 https://doi.org/10.1016/S0747-5632(02)00058-4 https://psycnet.apa.org/doi/10.1037/0022-0663.99.2.339 https://doi.org/10.1016/j.chb.2004.06.004 https://doi.org/10.1080/00220973.2015.1048847 https://doi.org/10.1109/MC.1987.1663693 https://doi.org/10.1016/S0022-5371(80)90312-6 https://doi.org/10.1016/S0022-5371(80)90312-6 https://doi.org/10.1016/j.chb.2005.08.012 https://doi.org/10.3200/JOER.99.3.156-166 http://www.psicothema.com/psicothema.asp?id=37 https://doi.org/10.1145/1059981.1059982 https://doi.org/10.1111/j.1467-9817.2012.01536.x https://doi.org/10.1111/j.1467-9817.2012.01536.x https://doi.org/10.1016/j.lindif.2017.09.001 https://doi.org/10.1016/j.chb.2016.02.058 https://doi.org/10.1007/BF02298109 Australasian Journal of Educational Technology, 2020, 36(4). 204 Kornmann, J., Kammerer, Y., Anjewierden, A., Zettler, I., Trautwein, U., & Gerjets, P. (2016). How children navigate a multiperspective hypermedia environment: The role of spatial working memory capacity. Computers in Human Behavior, 55(Part A), 145-158. https://doi.org/10.1016/j.chb.2015.08.054 Last, D., O’Donnell, A., & Kelly, E. A. (2001). The effects of prior knowledge and goal strength on the use of hypertext. Journal of Educational Multimedia and Hypermedia, 10(1), 3-25. Retrieved from https://www.learntechlib.org/primary/p/8447/ Lawless, K. A., & Kulikowich, J. M. (1996). Understanding hypertext navigation through cluster analysis. Journal of Educational Computing Research, 14(4), 385–399. https://doi.org/10.2190/DVAP-DE23-3XMV-9MXH Lee, M. J., & Tedder, M. C. (2003). The effect of three different computer texts on readers' recall: Based on working memory capacity. Computers in Human Behavior, 19(6), 767-783. https://doi.org/10.1016/S0747-5632(03)00008-6 Levin, J. R. (2008). The unmistakable professional promise of a young educational psychology researcher and scholar. Educational Psychologist, 43(2), 70-85. https://doi.org/10.1080/00461520801942276 Li, L.-Y., Tseng, S.-T., & Chen, G.-D. (2016). Effect of hypertext highlighting on browsing, reading, and navigational performance. Computers in Human Behavior, 54, 318-325. https://doi.org/10.1016/j.chb.2015.08.012 McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19(2), 113–139. https://doi.org/10.1007/s10648-006-9010-7 McCrudden, M. T., Schraw, G., & Kambe, G. (2005). The effect of relevance instructions on reading time and learning. Journal of Educational Psychology, 97(1), 88-102. https://doi.org/10.1037/0022- 0663.97.1.88 McDonald, S., & Stevenson, R. J. (1996). Disorientation in hypertext: The effects of three text structures on navigation performance. Applied Ergonomics 27(1), 61-68. https://doi.org/10.1016/0003- 6870(95)00073-9 Meyers, C. A., & Bagnall, R. G. (2015). A case study of an adult learner with ASD and ADHD in an undergraduate online learning environment. Australasian Journal of Educational Technology, 31(2), 208-219. https://doi.org/10.14742/ajet.1600 Müller-Kalthoff, T., & Möller, J. (2003). The effects of graphical overviews, prior knowledge, and self- concept on hypertext disorientation and learning achievement. Journal of Educational Multimedia and Hypermedia, 12(2), 117-134. Retrieved from https://www.learntechlib.org/primary/p/14576/ Nielsen, J. (2000). Designing web usability. Indianapolis, IN: New Riders Publishing. Nielsen, J. (2004). Guidelines for visualizing links. Retrieved from http://www.nngroup.com/articles/guidelines-for-visualizing-links/ Nielsen, J. (2008). Right-Justified navigation menus impede scannability. Retrieved from http://www.nngroup.com/articles/right-justified-navigation-menus Ozgungor, S., & Guthrie, J. T. (2004). Interactions among elaborative interrogation, knowledge, and interest in the process of constructing knowledge from text. Journal of Educational Psychology, 96(3), 437–443. https://doi.org/10.1037/0022-0663.96.3.437 Potelle, H., & Rouet, J.-F. (2003). Effects of content representation and readers' prior knowledge on the comprehension of hypertext. International Journal of Human-Computer Studies, 58(3), 327-345. https://doi.org/10.1016/S1071-5819(03)00016-8 Rezende, F., & de Souza Barros, S. (2008). Students' navigation patterns in the interaction with a mechanics hypermedia program. Computers & Education, 50(4), 1370-1382. https://doi.org/10.1016/j.compedu.2006.12.011 Rouet, J.-F. (2006). The skills of document use: From text comprehension to Web-based learning. Mahwah, NJ: Erlbaum.Schoon, P., & Cafolla, R. (2002). World Wide Web hypertext linkage patterns. Journal of Educational Multimedia and Hypermedia, 11(2), 117-139. Retrieved from https://www.learntechlib.org/primary/p/8958/ Shapiro, A., & Niederhauser, D. (2004). Learning from hypertext: Research issues and findings. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology, 2nd Edition (pp. 605–620). Mahwah, NJ: Erlbaum. Shih, Y.-C., Huang, P.-R., Hsu, Y.-C., & Chen, S. Y. (2012). A complete understanding of disorientation problems in web-based learning. The Turkish Online Journal of Educational Technology, 11(3), 1-13. Retrieved from https://www.learntechlib.org/p/55769/ https://doi.org/10.1016/j.chb.2015.08.054 https://www.learntechlib.org/primary/p/8447/ https://doi.org/10.2190/DVAP-DE23-3XMV-9MXH https://doi.org/10.1016/S0747-5632(03)00008-6 https://doi.org/10.1080/00461520801942276 https://doi.org/10.1016/j.chb.2015.08.012 https://doi.org/10.1007/s10648-006-9010-7 https://psycnet.apa.org/doi/10.1037/0022-0663.97.1.88 https://psycnet.apa.org/doi/10.1037/0022-0663.97.1.88 https://doi.org/10.1016/0003-6870(95)00073-9 https://doi.org/10.1016/0003-6870(95)00073-9 https://doi.org/10.14742/ajet.1600 https://www.learntechlib.org/primary/p/14576/ http://www.nngroup.com/articles/guidelines-for-visualizing-links/ http://www.nngroup.com/articles/right-justified-navigation-menus https://psycnet.apa.org/doi/10.1037/0022-0663.96.3.437 https://doi.org/10.1016/S1071-5819(03)00016-8 https://doi.org/10.1016/j.compedu.2006.12.011 https://www.learntechlib.org/primary/p/8958/ https://www.learntechlib.org/p/55769/ Australasian Journal of Educational Technology, 2020, 36(4). 205 Smith, B. L., Holliday, W. G., & Austin, H. W. (2010). Students' comprehension of science textbooks using a question-based reading strategy. Journal of Research in Science Teaching, 47(4), 363-379. https://doi.org/10.1002/tea.20378 Stanton, N. A. (1994). Explorations into hypertext: Spatial metaphor considered harmful. Innovations in Education & Training International 31(4), pp. 276-294. https://doi.org/10.1080/0954730940310404 Suchman, L. A. (1987). Plans and situated actions: A problem of human-machine communication. Cambridge, MA: Cambridge University Press. Thüring, M., Hannemann, J., & Haake, J. M. (1995). Hypermedia and cognition: Designing for comprehension. Communications of the ACM, 38(8), 57-66. https://doi.org/10.1145/208344.208348 van den Broek, P., Lorch, R. F. Jr., Linderholm, T., & Gustafson, M. (2001). The effects of readers’ goals on inference generation and memory for texts. Memory & Cognition, 29(8), 1081–1087. https://doi.org/10.3758/BF03206376 Vörös, Z., Rouet, J.-F., & Pléh, C. (2011). Effect of high-level content organizers on hypertext learning. Computers in Human Behavior, 27(5), 2047–2055. https://doi.org/10.1016/j.chb.2011.04.005 Corresponding author: Álvaro Jáñez, alvarojg@usal.es Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. Please cite as: Jáñez, Á., & Rosales, J.(2020). Novices’ performance using hypertext materials: Shedding light on disorientation. Australasian Journal of Educational Technology, 36(4), 189-205. https://doi.org/10.14742/ajet.4617 https://doi.org/10.1002/tea.20378 https://doi.org/10.1080/0954730940310404 https://doi.org/10.1145/208344.208348 https://doi.org/10.3758/BF03206376 https://doi.org/10.1016/j.chb.2011.04.005 mailto:alvarojg@usal.es https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.4617