rey.pdf Australasian Journal of Educational Technology 2011, 27(3), 397-410 Time advice and learning questions in computer simulations Günter Daniel Rey Julius-Maximilians-Universität Würzburg Students (N = 101) used an introductory text and a computer simulation to learn fundamental concepts about statistical analyses (e.g., analysis of variance, regression analysis and General Linear Model). Each learner was randomly assigned to one cell of a 2 (with or without time advice) x 3 (with learning questions and corrective feedback, with learning questions without feedback or without learning questions) between subjects factorial design. Time spent with the simulation as well as retention and transfer tests were used as dependent measures. Neither the time advice presented immediately before students chose to finish the simulation nor the learning questions presented during the simulation significantly improves learners' retention or transfer performances. Students who were asked to employ more time on the computer simulation or who received learning questions with corrective feedback spent significantly more time with the simulation than did students for whom the time advice or the learning questions were absent. The results were discussed on the basis of the cognitive theory of multimedia learning and the cognitive-affective theory of learning with media, as well as in conjunction with adaptive computer simulations. Introduction Research indicates that students develop misconceptions about important statistical themes, even with very good final marks in statistics (e.g., Castro Sotos, Vanhoof, Van den Noortgate & Onghena, 2007; Garfield & Ben-Zvi, 2007). For example, many students interpret the p-value as the strength of the effect, confuse the alpha error with the beta error, misinterpret confidence intervals or are even unable to understand the idea of the mean. Possibly, the use of computer simulations can partly overcome some of these problems and help to impart statistical knowledge to students. Computer simulations can be defined as programs where the user can perform experiments in controlled settings to understand how the underlying model of the simulation works (cf. de Jong & van Joolingen, 1998; van der Meij, 2007). These computer simulations typically have an underlying mathematical model programmed into them that dictates how the simulation behaves (Rieber, 2005). For example, a simulation that serves to understand the analysis of variance (ANOVA) and the regression analysis can be based on the General Linear Model (GLM). Learners can explore this underlying model by manipulating values of (input) variables and observing the behaviour of other (output) variables (de Jong, 2006). These simulations become increasingly important in learning statistics because they can be developed easily and are cost-efficient due to hardware and software improvements in recent years. However, simple implementation of computer simulations does not imply that learning effectiveness will be improved (de Jong, 2006). The purpose of the present 398 Australasian Journal of Educational Technology, 2011, 27(3) experiment was to investigate how computer simulations and instructional advice should be designed to optimise statistical knowledge transfer. In the following section, (a) the cognitive theory of multimedia learning (CTML) and the cognitive-affective theory of learning with media (CATLM) including the feedback principle, as well as (b) problems resulting from learning with computer simulations and (c) potential solutions to overcome these challenges are presented. An experiment is followed up to test two different solution approaches as well as a discussion containing theoretical and practical implications, limitations and future research directions. Cognitive theory of multimedia learning There are different general theories that can be applied in the context of learning with computer simulations like the extended scientific discovery as dual search (SDDS) model (van Joolingen & de Jong, 1997), the dual-coding theory (Paivio, 1986), the cognitive load theory (CLT, Sweller, 2005) or the cognitive theory of multimedia learning (CTML, Mayer, 2005a). The CTML (Mayer, 2005a) is based on the assumptions that the human information-processing system contains a visual/pictorial channel and an auditory/verbal channel (dual-channels assumption), each channel has a limited capacity for processing (limited capacity assumption), and active learning entails carrying out a coordinated set of cognitive processes during learning (active processing assumption). These assumptions are verified in innumerable experiments and are closely associated with Paivio's dual coding theory (Clark & Paivio, 1991; Paivio, 1986), Baddeley's model of working memory (Baddeley, 1992, 1999) and Sweller's CLT (Sweller, 1988, 2005). Furthermore, three different memory stores are assumed, which include sensory memory, working memory, and long-term memory. In these three memory stores, words and images are processed. The major cognitive processing required for learning with words and images are: selecting, organising and integrating. Selecting relevant words means that the learner is paying attention to some of the spoken or written words that are presented in the multimedia message as they pass through auditory sensory memory (Mayer, 2005a). Through this active process, a mental representation of selected words or phrases is created in the learner's verbal working memory. Selecting relevant images involves paying attention to static or dynamic pictures that are presented in the multimedia message as they pass through visual sensory memory (Mayer, 2005a). This process is also active and leads to a mental representation of selected pictures in the learner's visual working memory. Organising selected words refers to actively making connections between pieces of verbal knowledge. The output is a coherent verbal model of the selected words or phrases in the learner's working memory. For example, the organising process can lead to the construction of a representation of a causal connection. Organising selected images means that the learner actively makes connections among pieces of pictorial knowledge. The output is a coherent pictorial model of the selected images in the learner's working memory. For example, presented with a diagram, the learner may build a causal link in which one image representing the cause leads to a second image representing the effect. Integrating word based and image based representations refers to making connections between verbal and pictorial models as well as the learner's prior knowledge from long-term memory (Mayer, 2005a). Rey 399 Cognitive-affective theory of learning with media and the feedback principle The CATLM (Moreno, 2005; Moreno & Mayer, 2007) expands the CTML to media that present the learner with instructional materials other than words and pictures. While the CTML is based on three assumptions (dual-channels, limited capacity and active processing assumption), the CATLM uses these three plus four other assumptions. First, it is assumed that the long-term memory consists of a dynamic, evolving structure that holds both a memory for past experiences and a memory for general domain knowledge (cf. Tulving, 1977). Second, motivational factors are assumed to mediate learning by increasing or decreasing cognitive engagement (cf. Pintrich, 2003). Third, metacognitive factors are presumed to mediate learning by regulating cognitive processing and effect (cf. McGuinness, 1990). Fourth, it is assumed that differences in learners' prior knowledge and abilities may affect how much is learned with specific media (cf. Kalyuga, 2007). From the assumptions underlying a CATLM (respectively CTML), different cognitive principles of instructional design can be derived (Moreno & Mayer, 2007). One design principle is called the feedback principle. According to this principle students learn better with explanatory rather than with corrective feedback. Explanatory feedback consists of providing an explanation for why learners' answers are correct or not. This explanation is based on a principle. In contrast to explanatory feedback, corrective feedback consists of only communicating whether students' answers are correct or not. Empirical evidence for the feedback principle is supported by several studies (see Moreno & Mayer, 2007; cf. the questioning principle postulated from Campbell and Mayer, 2009). Problems resulting from learning with computer simulations Research indicates that many learners have substantial problems while learning with computer simulations (de Jong, 2006; de Jong & van Joolingen, 1998). For instance, Dunbar (1993) showed, in a simulation environment, that some students have a strong inclination to search for evidence that supports their current hypothesis, and that this inclination may prevent them from stating an alternative hypothesis, even when they are confronted with contradictory evidence. In a study from Eysink, Dijkstra and Kuper (2001) students experimented with a simulation designed for teaching first- order logic in a way that they were confronted with subject matter they already understood, but without confronting themselves with less familiar situations. Keselman (2003) demonstrated that learners varied too many variables at one time in a computer simulation about the multivariable risks of earthquakes. Therefore, these learners can not disentangle the effects of these different variables. In a study by Rey (2011, experiment 1) nearly 60% of the learners did not make use of a reset button at all in a computer simulation about self-organising maps, which is a special kind of neural network. The reset button served to reset the visualisation to its initial state and enabled a more systematic exploration of the simulation. Furthermore, learners fail to make predictions and make mistakes when interpreting data derived from the output of the simulation (Lewis, Stern & Linn, 1993). In summary, there are several empirical findings showing that learners often have serious problems learning adequately with computer simulations. 400 Australasian Journal of Educational Technology, 2011, 27(3) Potential solutions to overcome these problems There are different approaches to alleviate the aforementioned problems, guide learners in using computer simulations, and improve learning with these simulations. For example, assignments (i.e., exercises that set the simulation in the appropriate state), explanations and background information as well as monitoring tools and hypotheses scratchpads can be used to produce effective and efficient learning situations (de Jong, 2006). Consecutively, two different possible solutions leading to two hypotheses will be discussed in detail. First, adding time advice in computer simulations could improve students' learning outcome while learning with simulations. This approach is atypical to current methods postulated from other researchers which tried to improve the handling with the simulation (e.g., de Jong, 2006), but did not try to extend time spent with the simulation. It could be assumed that adding instructional advice on how to use the simulation could foster learning outcome at least partly by increasing the time spent with the computer simulation. In that case, only encouraging students to spend more time with the simulation should also improve learning outcome (e.g., by repeating the instructional material). Potentially, time advice might also help learners to invest more time in the three major cognitive processes required for learning with words and images (i.e., selecting, organising and integrating) postulated in the CTML and therefore helps build up an adequate mental representation of the computer simulation. Moreover, it could also be assumed that adding time advice can eliminate a possible production deficit (i.e., learners possess the required strategies for exploring the simulation adequately, but do not use these strategies spontaneously) in regard to exploring the simulation systematically, due to perceived time restrictions of the learners. Empirical evidence exists also showing that time advice can partly improve students' learning outcome while learning with simulations. In a study by Rey (2010) the presence of time advice was varied in a computer simulation about fundamental concepts in statistical analyses. Results revealed that the undergraduate students who were asked to employ more time on the simulation immediately before they chose to finish it spent considerably more time with the simulation and performed better on retention, but not on transfer, compared with students for whom this request was absent. The first hypothesis tries to replicate and extend the finding by Rey (2010) and therefore predicts that students who are asked to employ more time on the computer simulation while using the simulation spend more time with the simulation and perform better on retention and transfer than do students for whom this request is absent. As in Rey (2010), retention is defined as the ability to store information and retrieve or recognise the information later. This multidimensional ability is measured by testing, if learners can repeat, list, name, recognise or reproduce factual information (cf. Bloom & Krathwohl, 1956; Bloom, Madaus & Hastings, 1981; Mayer, 2005b). Contrary to retention, transfer performance is related to the multi-faceted potential to acquire the meaning of the stored information and apply it in new contexts. Second, adding learning questions in computer simulations could overcome the problem that learners often do not learn adequately with simulations (de Jong, 2006). Learning questions might encourage learners in generating predictions about the outcome of the computer simulation as well as collecting and evaluating evidence for their own prediction. On the one hand, empirical findings (e.g. Demetriadis, Rey 401 Papadopoulos, Stamelos & Fischer, 2008) for ill-structured domains suggest that question prompts can foster domain knowledge acquisition and knowledge transfer, whereas the effect is moderated by different aspects of the learning setting (e.g., whether time is restricted for processing the instructional materials, see Papadopoulos, Demetriadis, Stamelos & Tsoukalas, 2009). A quasi-experimental study (Ge & Land, 2003) measured four different student problem-solving processes (i.e., problem representation, generating solutions, making justifications, and monitoring and evaluating) in an ill-structured task in the context of a lecture session. Students who received questions prompts in printed format or through the Internet outperformed those who received no prompts on all four dependent measures. On the other hand, empirical findings also support the assumption that adding learning questions does not improve learning outcome (e.g., Papadopoulos, et al., 2009). Greene and Land (2000) found that questions prompting was often insufficient as a scaffold because students sometimes omitted questions or answered superficially, thereby failing to engage in deeper processing (see also Davis & Linn, 2000; Ge & Land, 2003, 2004). In a study by Rey (2010), students’ retention and transfer performance did not improve by adding the learning questions without feedback presented during a computer simulation about fundamental concepts in statistical analyses. The author assumes that learners could benefit from learning questions, but only with corrective or explanatory feedback (cf. the feedback principle in the CATLM). Therefore, the present experiment tries to expand the effort by Rey (2010) and test if corrective feedback is really sufficient to improve students’ learning outcomes. Therefore, the second hypothesis predicts that students who are given learning questions with corrective feedback about the content of the computer simulation while using the simulation, perform better on retention and transfer than do students for whom these questions or the corrective feedback are absent. Method Participants and design The participants were 101 students recruited from the University of Trier (Germany). The students took part in the experiment to fulfill test subject hours or in order to participate in a seminar concerning data analysis with Excel. The mean age of the participants was 23.1 (SD = 3.8) years and the overall percentage of women was 68.3%. Most (94.1%) of the students were enrolled in psychology as their main subject. Each student was randomly assigned to one of the six treatment groups (2 x 3). The computer-presented material was exactly the same as that used in the experiment by Rey (2010). It differed only in the between-subject factors manipulated. The material consisted of an illustrated introductory text and a computer simulation about the ANOVA, regression analysis and the GLM, retention and transfer tests, as well as a questionnaire about the perceived quality of the instructional material. The simulation presented two different dynamic visualisations, where the learner could change different parameters with scrollbars and radio buttons and observe the resulting effects (see Figure 1). Manipulating the different parameters resulted in a modified visualisation immediately. Learners could also change the perspective or rotate the visualisations by using scrollbars. The time advice factor (with or without a request to employ more time on the computer simulation) was the first between-subject factor manipulated. The advice 402 Australasian Journal of Educational Technology, 2011, 27(3) was "Dear participant, we would like to advise you to look at the interactive visualisation a little bit longer". It was presented during the computer simulation after the participant in the time advice condition pressed the button to finish the computer simulation. Instead of finishing, the advice appeared. If the participant pressed the button to finish the computer simulation a second time, the same time advice appeared again. Pressing the button to finish the simulation for the third time, the participant actually finished the simulation. Every participant in the time advice condition received the time advice two times, independently of time they actually spent with the simulation before the advice. There were 51 participants in the group with time advice and 50 subjects in the group without time advice. Figure 1: Selected frame from the group with learning questions (with or without feedback). The learning question button was placed in the right upper corner ("Beispielfragen aufrufen"). The learning questions factor (with learning questions and feedback, with learning questions without feedback or without learning questions) was the second between- subject factor manipulated. Two learning questions appeared if the student pressed the button labeled "Show example questions". This button had to be pressed at least once to finish the computer simulation. The button was placed in the right upper corner (see Figure 1). After pressing the button, two example questions about the content of the computer simulation appeared in multiple choice format (six response options per question where only one answer per question was correct). Both questions were displayed in an extra window which could be closed and opened as often as required. The response options could be arbitrarily often re-chosen by the students. One question was a retention question, the other a transfer question. In both cases, students in the condition with learning questions and feedback received the feedback “Richtig!” Rey 403 (“Correct!”) if they had chosen the correct response option. This text feedback was presented right behind the text of the response option in bold face. Students in the condition with learning questions without feedback received no feedback after answering the question (i.e., choosing a radio button). Students in the condition without learning questions received no learning questions at all. Neither question was used in the subsequent retention and transfer test. There were 34 participants in the group with questions and feedback, 33 participants in the group with questions and without feedback and 34 subjects in the group without questions. Comparisons were made between the six groups (2 x 3) on measures of retention, transfer and time spent with the simulation. Instrument and materials – tasks The multimedia presentations were developed using Microsoft Excel and Visual Basic Application (VBA) and are only available in the German language. For each participant, the computer materials contained the instructional material, the retention and transfer tests, a short questionnaire about the perceived quality of the instructional material, as well as a participant questionnaire soliciting information concerning the student’s gender, age, field of study, number of terms, and self evaluated general computer knowledge. Time was measured separately for the introductory text, the simulation, as well as the retention and transfer tests. Click frequency for all interactive elements (scrollbar and buttons) was recorded as well. The illustrated text in the fore field of the computer simulation consisted of nine pages containing approximately 2350 words as well as one table and five figures about the ANOVA and regression analysis. The text gave a short introduction to these statistical analyses and described how both are congruent with each other in the context of the GLM. The text also pointed to the overfitting problem in statistical analyses and explained the functionality of the subsequent presented computer simulation in detail with two additional figures. The simulation consisted of three components. First, in the upper area of the screen ("Parametereinstellungen"), different parameters with scrollbars and radio buttons could be modified by the learner (see Figure 1). There, students could change the effect sizes ("Effektgrößen") for the two main effects and the interaction between the two predictor variables, choose between a linear, quadratic or cubic model ("Modellterm") and observe the explained variance of the model ("Varianzaufklärung"). Furthermore, students could rotate the figures and alter the perspective ("3D-Ansicht") amongst others and perform a cross-validation ("Kreuzvalidierung"). Second, manipulating the different parameters (e.g., the main effect for the predictor variable A) immediately changed the visualisation on the left side, which represented the predicted dataset ("vorhergesagte Daten") in a three-dimensional coordinate system (see Figure 1). Third, the visualisation on the right side, representing a fictitious empirical dataset ("'empirische' Daten"), could be modified by testing the resulting prediction on a new dataset in order to avoid overfitting. Both visualisations were changed simultaneously by rotating the figures or by altering the perspective. Retention is the ability to store information and retrieve or recognise the information later. The retention test consisted of ten multiple choice questions. Each question included five or six response options where only one answer per question was correct. As an example, the question "Which effect was presented on the x-axis in the 404 Australasian Journal of Educational Technology, 2011, 27(3) interactive visualisation?" contained five different response options, for instance, "Main effect A". Another question asked what model terms can be found in the visualisation. It included six different response options, for instance, a linear model term. All retention questions could be answered with the information that was given in the simulation without the inference of additional information. Contrary to retention, transfer performance is related to the multi-faceted potential to acquire the meaning of the stored information and apply it in new contexts. The transfer test consisted of nine multiple choice questions (four to six response options where only one answer per question was correct) and one questions in open response format. For this question ("How big was the maximum explained variance that could be reached in the master sample?"), participants had to insert a number in a text box. A multiple choice transfer question asked, for instance, what does a plane (or respectively, parallel lines) in the right graph of the interactive visualisation imply for the underlying effects. In all transfer questions inferences had to be drawn from the presented information in the visualisation (cf. Bloom & Krathwohl, 1956; Bloom, et al., 1981; Mayer, 2005b). For example, the interactive visualisation in the right graph did not show a plane or parallel lines. All retention and transfer questions were created by an expert for statistical analysis and on the basis of the taxonomy of learning from Bloom (e.g. Bloom & Krathwohl, 1956; Bloom, et al., 1981). The questionnaire (Cronbach’s alpha = .82) about the perceived quality of the instructional material consisted of five questions, all containing 7-point Likert scales (the ‘more the merrier’ was the perceived quality). Two questions asked how useful the dynamic visualisation was for understanding the underlying concepts and how fast the underlying concepts in the dynamic visualisation could be comprehended. In the three remaining questions, learners should judge the entire instructional material as well as the dynamic visualisation only and they had to evaluate the didactical quality of the dynamic visualisation. Procedure and scoring Participants were tested in groups of 1 to 20 per session. Each student was randomly assigned to one of the six treatment groups (2 x 3) and was seated at an individual cubicle in front of a computer. The participants completed the entire experiment at their own rate and without any time limit. On average, they spent about 20-40 minutes with the instructional text and the simulation. On the introductory page, students were welcomed and thanked for participating in the experiment as well as advised to study the instructional materials carefully and to use and pay attention to the simulation precisely and extensively. While working with the simulation students no longer had access to the introductory text. After finishing the simulation participants answered the same retention and transfer questions as well as the same questionnaire about the perceived quality of the instructional material without having access to the simulation anymore. Finally, participants were thanked for their participation and debriefed. For the multiple choice retention and transfer questions, the participants received one point for choosing the correct response option. For the only transfer question with open response format, the subjects could gain an additional transfer point if the inserted number in the text box was within a predefined accepted narrow value range. The correct answer and score assessment for this question was defined very precisely before the study, so two independent raters agreed on 100% of this open response Rey 405 format transfer question. The final score was reached by adding together the student's scores on each individual retention and each individual transfer question. Therefore, a maximum score of ten points for retention and ten points for transfer could be achieved. The final score for the evaluation of the instructional material was reached by averaging each student’s score on the five questions (for each question one to seven points). Hence, a minimum score of one point and a maximum score of seven points could be reached. Results Table 1 shows the mean scores and standard deviations for the six different groups on measures of retention, transfer and time spent with the computer simulation (without the instructional text and the retention and transfer questions). A two-factor multivariate analysis of variance (MANOVA) was conducted, with time advice (with a request to employ more time on the computer simulation vs. without such a request) and learning questions (with learning questions and feedback vs. with learning questions without feedback vs. without learning questions) as between-subjects factors, and retention, transfer and time spent with the simulation as dependent measures. The assumption of homogeneity of variance was tested prior to the MANOVA and found to be tenable, Box’s M (30, 20312.1) = 45.02, p = .08. Two-way ANOVAs on each dependent variable were conducted as follow-up tests to the MANOVA. Table 1: Mean score on retention and transfer tests and time spent with the computer simulation (in minutes and seconds without the instructional text and without the retention and transfer questions) and their corresponding standard deviations for the six different groups (2 x 3) Type of measureGroup Retention Transfer Time Time advice Learning questions Group size M SD M SD M SD – + and feedback 17 6.76 1.30 6.00 1.73 15:07 7:13 – + without feedback 16 6.56 1.59 5.69 1.82 10:23 4:34 – – 17 6.59 1.73 4.65 1.84 8:53 5:08 + + and feedback 17 6.82 1.98 5.59 2.24 15:48 8:06 + + without feedback 17 7.06 2.22 5.47 2.07 17:04 11:33 + – 17 6.76 1.56 5.12 2.03 11:41 5:57 Note: "+" means "with"; "–" means "without". Potential scores ranged from 0 to 10 for the retention and the transfer score. Times in minutes:seconds. Time advice Hypothesis 1: Students who are asked to employ more time on the computer simulation while using the simulation spend more time with the simulation and perform better on retention and transfer than do students for whom this request is absent. Students who were asked to employ more time on the computer simulation spent over three minutes (3 minutes and 22.0 seconds) more time with the simulation (M = 14 minutes and 50.8 seconds, SD = 8 minutes and 57.7 seconds) than did students for whom this request was absent (M = 11 minutes and 28.8 seconds, SD = 6 minutes and 16.4 seconds), F(1, 95) = 5.18, p = .03. The effect size (Cohen's d) was .44 on time, indicating a medium effect size. 406 Australasian Journal of Educational Technology, 2011, 27(3) No significant effects were found on the retention test, F(1, 95) < 1, p = .49, or on the transfer test, F(1, 95) < 1, p = .89. Statistically, the null hypothesis could be accepted for an effect size of f 2 = .15 due to the sufficient power (1-beta = .97 for alpha = .05). The effect sizes (Cohen's d) were .14 on retention and -.02 on transfer, indicating very small effect sizes. Overall, these results show that learners who are asked to employ more time on the computer simulation while using the simulation spend more time with the simulation, but do not perform better on retention or transfer than do students for whom this request is absent. Learning questions Hypothesis 2: Students who are given learning questions with corrective feedback about the content of the computer simulation while using the simulation perform better on retention and transfer than do students for whom these questions or the corrective feedback are absent. No significant effects were found on the retention test, F(2, 95) < 1, p = .94, or on the transfer test, F(2, 95) = 2.00, p = .14. Statistically, the null hypothesis could be accepted for an effect size of f 2 = .15 due to the sufficient power (1-beta = .94 for alpha = .05). The effect sizes (partial eta squared) were .001 on retention and .04 on transfer, indicating small to medium effect sizes. In contrast to retention and transfer performance, time spent with the computer simulation reached significance, F(2, 95) = 4.21, p = .02. Post-hoc tests (Tukey-HSD) revealed that students receiving learning questions with corrective feedback spent over five minutes (5 minutes and 10.4 seconds) more time with the simulation (M = 15 minutes and 27.3 seconds, S D = 7 minutes and 33.7 seconds) than students not receiving learning questions (M = 10 minutes and 17.0 seconds, SD = 5 minutes and 39.4 seconds), p = .01. Overall, these results do not show that learners who are given learning questions with corrective feedback about the content of the computer simulation while using the simulation perform better on retention and transfer than do students for whom these questions or the corrective feedback are absent. However, learners who are given learning questions with corrective feedback spend more time with the simulation than do learners for whom these learning questions are absent. Further findings There was no significant interaction among two between-subjects factors (time advice and learning questions) on the dependent measures (.26 ≤ p ≤ .87). Statistically, the null hypotheses could be accepted for an effect size of f2 = .15 due to the sufficient power (1- beta = .94 for alpha = .05). The effect size (partial eta squared) was between 0.003 and 0.03, indicating small effect sizes. No significant differences were found for the two between-subjects factors as well as their interaction in regard to the questionnaire about the perceived quality of the instructional material (.11 ≤ p ≤ .91). Furthermore, a two-factor MANOVA was conducted, with time advice (with a request to employ more time on the computer simulation vs. without such a request) and learning questions (with learning questions and feedback vs. with learning questions without feedback vs. without learning questions) as between-subjects factors, and click frequencies for all scrollbars and radio buttons in the simulation as dependent measures. Neither the main effect for time advice (Wilk’s lambda = .83), F(14, 82) = Rey 407 1.21, p = .29, partial eta squared = .17, nor the main effect for learning questions (Wilk’s lambda = .67), F(28, 164) = 1.31, p = .16, partial eta squared = .18, reached significance. In addition, the interaction between these two factors also failed to reach significance, (Wilk’s lambda = .67), F(28, 164) = 1.30, p = .16, partial eta squared = .18. Discussion The goal of this research was to investigate whether time advice and learning questions (with or without corrective feedback) in a computer simulation affect learning and time spent with the simulation. Learners who were asked to employ more time on the computer simulation while using the simulation spent more time with the simulation, but did not perform better on retention or transfer than did students for whom this request was absent. Therefore, the present study only partly replicates the finding by Rey (2010). Potentially, learners in the present study did not adequately repeat the contents of the simulation while working longer on the simulation through the time advice. In addition, the time advice seems not to help learners to invest more time in the three major cognitive processes required for learning with words and images (i.e., selecting, organising and integrating) postulated in the CTML and therefore assumedly also does not help build up an adequate mental representation of the computer simulation in the present study. Furthermore, students possibly did suffer a mediation deficit (i.e., learners do not yet possess the required strategies for exploring the simulation adequately) rather than a production deficit (i.e., learners possess the required strategies for exploring the simulation adequately, but do not use these strategies spontaneously) in the present study. Learning questions (with or without corrective feedback) about the content of the computer simulation presented while using the simulation did not influence learning performance (i.e. retention and transfer performance), but influenced the time spent with the simulation. Learners who were given learning questions with corrective feedback spent more time with the simulation than did learners for whom these learning questions were absent. The present study does not support the assumption by Rey (2010) that learners could benefit from learning questions with corrective feedback. In contrast, the present result indicates that simple corrective feedback seems to be insufficient to improve the learning outcome. However, the current findings did not test and therefore did not contradict the feedback principle of the CATLM, which postulates that students learn better with explanatory rather than with corrective feedback. Analysing click frequencies and the questionnaire about the perceived quality of the instructional material did not reveal any significant differences for the different instructional conditions. Overall, neither time advice nor learning questions influenced learning outcome, but influenced learner's time spent with the computer simulation. However, spending more time with the simulation seems not to imply deeper understanding of the simulation. Instead, it can be assumed that most of the learners still do not know how to use the simulation adequately and therefore do not benefit from time advice and learning questions with regard to learning performance. Implications and limitations On the theoretical side, the present findings can be embedded in the CTML and the CATLM. First, it can be assumed that simple time advice and learning questions with only corrective feedback are not sufficient to alleviate the problems of learning 408 Australasian Journal of Educational Technology, 2011, 27(3) adequately with computer simulations. More precisely, the three major cognitive processes required for learning with words and images (i.e., selecting, organising and integrating) seem not to profit from time advice and learning questions with corrective feedback. These supporting measures might not be sufficient to help learners build up an adequate mental representation of the computer simulation. Second, the present findings for learning questions might be tied to the feedback principle of the CATLM, postulating that students learn better with explanatory rather than with corrective feedback. On the practical side, the recent findings emphasise that learners often have serious problems learning adequately with computer simulations. Simple time advice and learning questions with only corrective feedback are not always successful to alleviate this problem. Possibly, the learning questions should contain explanatory feedback rather than only corrective feedback. The present study was limited by the short-term nature of the instructional materials (i.e., they consisted of only 20-40 minutes of concentrated instruction) and the limited genre of the instructional materials (i.e., an introductory text and a computer simulation about the analysis of variance, the regression analysis and the General Linear Model). The ability to generalise was also limited by the nature of the test (i.e., primarily multiple choice questions created by an expert for statistical analysis and given immediately after instruction) as well as the non-authentic context (i.e., as a required psychology experiment). Subsequent research is needed to determine whether the same pattern of findings would occur for other instructional materials and other contexts. Future directions Prospectively, computer simulations that are not only interactive, but at the same time adaptive, will be used more frequently in statistical learning (cf. van Merriënboer & Sweller, 2005). Adaptive computer simulations are simulations that react to parameter changes implemented by the user and assess the learner's behaviour. This assessment serves as a basis to modify the visualisation or to give personalised feedback to the learner. There are different kinds of user behaviour, which can be assessed. For example, the learner's way of using the different interactive elements in the simulation can be assessed (e.g., based on log files of the student’s interaction, see e.g., Veermans, van Joolingen & de Jong, 2006), followed by personalised instructional or time advice to improve the learner's utilisation and to attain a deeper understanding of the presented instructional material (cf. Lin & Lehman, 1999). Overall, future studies should investigate the implications of adaptive computer simulations and adaptive learning environments for human information processing and effects for learning outcome instead of merely focusing on their technical implementation, or tailoring instructional content to relatively superficial learner attributes (Kalyuga, 2008). References Baddeley, A. D. (1992). Working memory. Science, 255, 556-559. Baddeley, A. D. (1999). Human memory. Boston: Allyn & Bacon. Bloom, B. S. & Krathwohl, D. R. (1956). Taxonomy of educational objectives. The classification of educational goals, Handbook I: Cognitive domain. New York: Longmans Green. Rey 409 Bloom, B. S., Madaus, G. F. & Hastings, J. T. (1981). Evaluation to improve learning. New York, NY: McGraw-Hill. Campbell, J. & Mayer, R. E. (2009). Questioning as an instructional method: Does it affect learning from lectures? Applied Cognitive Psychology, 23, 747-759. Castro Sotos, A. E., Vanhoof, S., Van den Noortgate, W. & Onghena, P. (2007). Students' misconceptions of statistical inference: A review of the empirical evidence from research on statistics education. Educational Research Review, 2, 98-113. Clark, J. M. & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3, 149-210. Davis, E. A. & Linn, M. (2000). Scaffolding students' knowledge integration: Prompts for reflection in KIE. International Journal of Science Education, 22, 819-837. de Jong, T. (2006). Computer simulations - Technological advances in inquiry learning. Science, 312, 532-533. de Jong, T. & van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68, 179-201. Demetriadis, S. N., Papadopoulos, P. M., Stamelos, I. G. & Fischer, F. (2008). The effect of scaffolding students' context-generating cognitive activity in technology-enhanced case- based learning. Computers & Education, 51, 939-954. http://dx.doi.org/10.1016/j.compedu.2007.09.012 Dunbar, K. (1993). Concept discovery in a scientific domain. Cognitive Science, 17, 397-434. Eysink, T. H. S., Dijkstra, S. & Kuper, J. (2001). Cognitive processes in solving variants of computer-based problems used in logic teaching. Computers in Human Behavior, 17, 1-19. Garfield, J. & Ben-Zvi, D. (2007). How students learn statistics revisited: A current review of research on teaching and learning statistics. International Statistical Review, 75, 372-396. Ge, X. & Land, S. M. (2003). Scaffolding students' problem-solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology, Research and Development, 51, 21-38. Ge, X. & Land, S. M. (2004). A conceptual framework for scaffolding ill-structured problem- solving processes-using question prompts and peer interactions. Educational Technology, Research and Development, 52, 5-22. Greene, B. A. & Land, S. M. (2000). A qualitative analysis of scaffolding use in a resource-based learning environment involving the World Wide Web. Journal of Educational Computing Research, 23, 151-179. Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review, 19, 509-539. Kalyuga, S. (2008). When less is more in cognitive diagnosis: a rapid online method for diagnosing learner task-specific expertise. Journal of Educational Psychology, 100, 603-612. Keselman, A. (2003). Supporting inquiry learning by promoting normative understanding of multivariable causality. Journal of Research in Science Teaching, 40, 898-921. Lewis, E. L., Stern, J. L. & Linn, M. C. (1993). The effect of computer simulations on introductory thermodynamics understanding. Educational Technology, 33, 45-58. Lin, X. & Lehman, J. D. (1999). Supporting learning of variable control in a computer-based biology environment: Effects of prompting college students to reflect on their own thinking. Journal of Research in Science Teaching, 36, 837-858. Mayer, R. E. (2005a). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31-48). Cambridge, MA: CUP. Mayer, R. E. (2005b). Introduction to multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 1-16). Cambridge, MA: Cambridge University Press. 410 Australasian Journal of Educational Technology, 2011, 27(3) McGuinness, C. (1990). Talking about thinking: The role of metacognition in teaching thinking. In K. Gilhooly, M. Deane & G. Erdos (Eds.), Lines of thinking (Vol. 2, pp. 310-312). San Diego: Academic. Moreno, R. (2005). Instructional technology: Promise and pitfalls. In L. Pytlikzillig, M. Bodvarsson & R. Bruning (Eds.), Technology-based education: Bringing researchers and practitioners together (pp. 1-19). Greenwich, CT: Information Age Publishing. Moreno, R. & Mayer, R. (2007). Interactive multimodal learning environments: Special issue on interactive learning environments: Contemporary issues and trends. Educational Psychology Review, 19, 309-326. Paivio, A. (1986). Mental representations: A dual coding approach. New York: OUP. Papadopoulos, P. M., Demetriadis, S. N., Stamelos, I. G. & Tsoukalas, I. A. (2009). Prompting students’ context-generating cognitive activity in ill-structured domains: Does the prompting mode affect learning? Educational Technology, Research and Development, 57, 193-210. Pintrich, P. R. (2003). Motivation and classroom learning. In W. M. Reynolds & G. E. Miller (Eds.), Handbook of psychology: Educational psychology (pp. 103-122). New York: Wiley. Rey, G. D. (2010). Instructional advice, time advice and learning questions in computer simulations. Australasian Journal of Educational Technology, 26(5), 675-689. http://www.ascilite.org.au/ajet/ajet26/rey.html Rey, G. D. (2011). Reset button and instructional advice in computer simulations. European Psychologist, 16, 58-67. Rieber, L. P. (2005). Multimedia learning in games, simulations, and microworlds. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 549-567). Cambridge, MA: Cambridge University Press. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285. Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 19-30). Cambridge, MA: Cambridge University Press. Tulving, E. (1977). Episodic and semantic memory. In E. Tulving & W. Donaldson (Eds.), Organization of memory (pp. 381-403). New York: Academic Press. van der Meij, J. (2007). Support for learning with multiple representations: Designing simulation-based learning environments. University of Twente, Enschede, The Netherlands. van Joolingen, W. R. & de Jong, M. T. (1997). An extended dual search space model of scientific discovery learning. Instructional Science, 25, 307-346. van Merriënboer, J. J. G. & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147-177. Veermans, K., van Joolingen, W. R. & de Jong, T. (2006). Use of heuristics to facilitate scientific discovery learning in a simulation learning environment in a physics domain. International Journal of Science Education, 28, 341-361. Author: Günter Daniel Rey PhD, Institute for Psychology, Department Psychology IV Julius-Maximilians-Universität Würzburg, Röntgenring 10, 97070 Würzburg, Germany Email: rey@psychologie.uni-wuerzburg.de Web: http://www.i4.psychologie.uni-wuerzburg.de/es/entwicklungspsychologie/ mitarbeiter/mitarbeiter/dr_guenter_daniel_rey/ Please cite as: Rey, G. D. (2010). Time advice and learning questions in computer simulations. Australasian Journal of Educational Technology, 27(3), 397-410. http://www.ascilite.org.au/ajet/ajet27/rey.html