Original Research With a little help. . . : On the role of guidance in the acquisition and utilisation of knowledge in the control of complex, dynamic systems Natassia Goode1 and Jens F. Beckmann2 1Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Australia and 2School of Education, Durham University, Durham, UK Many situations require people to acquire knowledge about, and learn how to control, complex dynamic sys- tems of inter-connected variables. Numerous studies have found that most problem solvers are unable to ac- quire complete knowledge of the underlying structure of a system through an unguided exploration of the system variables; additional instruction or guidance is required. This paper examines whether providing structural infor- mation following an unguided exploration also improves control performance, and the extent to which any im- provements are moderated by problem solvers’ fluid intel- ligence as measured via Raven’s APM. A sample of 98 participants attempted to discover the underlying struc- ture of a computer-simulated complex dynamic system. After initially controlling the system with their indepen- dently acquired knowledge, half of the sample received in- formation and an explanation of the underlying structure of the system. All participants then controlled the system again. In contrast to the results of previous studies, the provided information resulted in immediate improvements in control performance. Fluid intelligence as measured via APM moderated the extent to which participants bene- fited from the intervention. These results indicate that guidance in the form of structural information is critical in facilitating knowledge acquisition and subsequent use or application of such knowledge when controlling com- plex and dynamic systems. Keywords: complex problem solving, dynamic systems, knowledge acquisition, fluid intelligence, discovery learning Many situations require us to acquire knowledgeabout, and learn to control, dynamic systems of causally connected variables. Learning how to heat food in a microwave, respond to emails and buy train tickets are just a few of the many examples that might be encountered in everyday life. A significant body of research has examined the conditions that facilitate the acquisition of knowledge about complex and dy- namic systems (de Jong & van Joolingen, 1998; de Jong, Linn, & Zacharia, 2013). A question that has been addressed less frequently is how is knowledge best acquired to most effectively control such systems? This paper examines whether structural information (i.e., an explanation of how each input affects each output with a diagram that depicts the system vari- ables, the direction and strengths of their interrela- tion) confers any advantage over an unguided explo- ration of the system, its variables and their intercon- nectedness. We also investigate the role of fluid intel- ligence as measured via Raven’s Advanced Progressive Matrices (APM; Raven, Raven, & Court, 1998) in util- ising this information. The complex problem solving (CPS) approach To investigate how people learn how to control com- plex and dynamic systems in the real world, a wide variety of computer-based problem-solving scenarios have been developed (e.g. Berry & Broadbent, 1984; Dörner, 1980; Funke, 1992; for a review see Osman, 2010). The study presented in this article was under- pinned by the DYNAMIS or complex problem solving (CPS) approach, introduced by Funke (1992; 2001; see Blech & Funke, 2005 for a review). CPS tasks con- sist of a number of inputs (variables that the problem solver intervenes on) and outputs (outcomes that are generated by the system) that are governed by a set of linear equations (this is referred to as the underlying structure of the system). The systems are dynamic in the sense that the current output value depends on the value of the input selected by the problem solver, and the previous value of the output. Some CPS tasks also include autonomic changes, so that the values of par- ticular output variables change independently on each trial. A typical experimental procedure using this ap- proach consists of an initial exploration phase in which problem solvers are required to diagnose the underly- ing structure of the system. In a subsequent control phase they are instructed to control the system by ma- nipulating the input variables to reach and maintain specific goal values of the output variables. This means that separate measures of structural knowledge and control performance can be derived, and the cognitive processes of knowledge acquisition and knowledge ap- plication can be studied independently. Corresponding author: : Dr. Natassia Goode, Centre for Human Factors and Sociotechnical Systems, Faculty of Arts, Business and Law, University of the Sunshine Coast, Locked Bag 4, Maroochydore DC QLD 4558, Australia. e-mail: ngoode@usc.edu.au 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 1 mailto:ngoode@usc.edu.au https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems The role of structural knowledge in controlling a complex, dynamic system The acquisition of knowledge through an unguided ex- ploration of a system and its interrelated variables can be characterised as discovery (De Jong & van Joolin- gen, 1997; 1998) or inquiry-based learning (Lazonder & Harmsen, 2016). In this approach, the learner is seen as an independent and active agent in the pro- cess of knowledge acquisition, as they must develop hypotheses, design experiments to test them, and ap- propriately interpret the data (De Jong & van Joolin- gen, 1998). In educational settings, the problems that learn- ers experience with unguided inquiry-based learning are well documented. A recent meta-analysis of 164 studies found that across domains, unguided inquiry- based learning is less effective than explicit instruc- tion for acquiring knowledge. However, the advantage is reversed when learners receive adequate guidance during inquiry-based learning; they learn more than those taught using explicit instruction (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011). Numerous studies in educational settings have also found that learners need at least some guidance during exploration in order to facilitate the acquisition of complete structural knowl- edge (De Jong & van Joolingen, 1998; De Jong, 2005, 2006; Kirschner, Sweller & Clark, 2006; Lazonder & Harmsen, 2016; Mayer, 2004). Similarly, in research with CPS tasks, it has been found that most problem solvers are unable to acquire a complete or accurate representation of the underly- ing structure of the system through an unguided explo- ration of the system variables (Beckmann, 1994; Beck- man & Guthke, 1995; Burns & Vollmeyer, 2002; Funke & Müller, 1988; Kluge, 2008; Kröner, 2001; Kröner, Plass & Leutner, 2005; Müller, 1993; Osman, 2008; Schoppek, 2002; Vollmeyer, Burns, & Holyoak, 1996). These studies also report a consistent positive relation- ship between the amount of structural knowledge that is acquired and the quality of problem solvers’ control performance (see Knowledge Hypothesis). It is worth noting that the majority of these studies are correlational and therefore do not allow for causal interpretations of the reported association between knowledge and control performance. The study of Goode and Beckmann (2010) is one of the rather rare examples where an experimental design was adopted to test the causal nature of this association. They found that control performance improved systemati- cally as the amount of structural information available to participants increased, and that at least some struc- tural knowledge was required to perform better than simulated random control interventions. This study illustrates that control performance is causally depen- dent on the amount of knowledge that is acquired about the underlying structure of the system. The impact of providing structural information on control performance In this study we are interested in determining whether supplementing problem solvers exploration of a CPS task with structural information results in better con- trol performance than an unguided exploration of the system variables. A study conducted by Süß (1996, p. 166-177) sug- gested that providing structural information benefits structural knowledge, but confers no advantage for control performance. This study used a dynamic decision-making task called "TAILORSHOP", which is intended to simulate a small business that produces and sells shirts. The system consists of 24 variables inter-connected by 38 relations. The values of twenty- one of these variables are represented on the user- interface, and three are invisible. Twelve of the vari- ables can be manipulated directly by participants, and the goal is to increase the value of the variable "com- pany value" (Danner et al., 2011). The underlying structure is intended to reflect problem solvers’ prior knowledge of similar "real world" scenarios (see Beck- mann & Goode, 2014 for a discussion of the problems associated with this assumption). In Süß’s (1996) study, one group of participants ex- plored "TAILORSHOP" while another group studied a causal diagram for the same time period, before both performing the control task. A control group performed the control task without any prior explo- ration or intervention. Structural knowledge was as- sessed prior to, and after, interacting with the task. The group that studied the causal diagram acquired more structural knowledge than the exploration group; there was no difference between the exploration and control group. Surprisingly, there was no differences in control performance across the conditions. Thus, the causal diagram appeared to benefit structural knowl- edge but not control performance. However, these re- sults should be interpreted with caution, as a strong associative link was found between structural knowl- edge prior to interacting with the task and control per- formance across all conditions. On the one hand, this could be interpreted as an indicator of the "ecological validity" of the system. On the other hand, the al- ready substantial correlation between prior knowledge and control performance limits the potential impact of the interventions (i.e. knowledge acquisition through causal diagram or exploration), and introduces a po- tential source of individual differences among partici- pants. A number of studies using abstract systems (Putz- Osterloh, 1993; Preußler, 1996, 1998) also report sim- ilar results to Süß (1996). These studies suggest that problem solvers require a period of active practice applying structural information before they demon- strate an advantage over knowledge acquired through unguided exploration (Putz-Osterloh, 1993; Preußler, 1996; Preußler, 1998). These studies used the CPS task "LINAS", which contains four inputs and seven outputs interconnected by fifteen linear relations. The 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 2 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems labels given to the system variables did not refer to objects in the real world (e.g. "Bulmin", "Ordal", "Tri- mol") to control for the influence of prior knowledge. In Putz-Osterloh’s (1993) study an experimental group (given a causal diagram) and a control group were first instructed to diagnose the underlying structure of a system by exploring the system variables. The causal diagram illustrated the input and output variables as rectangles linked by arrows to indicate the relation- ships between them; the meaning of the diagram was verbally explained by the experimenter. Against ex- pectations, the experimental group performed no bet- ter than the control group in a subsequent control task. However, in a follow-up study six months later, the ex- perimental group had better control performance than the control group. Given that the advantage to per- formance was only evident after participants had con- siderable exposure to the task, Putz-Osterloh (1993) suggested that problem solvers might need a period of practice applying their knowledge in order to ben- efit from structural information. However, caution is urged in interpreting these findings due to the rela- tively small sample size (N = 16 - 25 per condition). Putz-Osterloh’s (1993) interpretation of her find- ings found further support in a series of studies con- ducted by Preußler (Preußler, 1996, 1998). In the first experiment, participants in an experimental group were instructed using standardised examples as to how each input affected each output; a control group ex- plored the system without assistance. No differences in control performance were found. In line with Putz- Osterloh’s (1993) study, it was argued that the struc- tural information did not provide an advantage to con- trol performance because participants did not have a chance to practice applying it (Preußler, 1996). There- fore, in a later experiment, Preußler (1998) gave an experimental group a causal diagram, and in addition they completed practice tasks in which goal values had to be attained by manipulating the input vari- ables. Each task was repeated until the problem solver reached the target values. The control group had to perform the same practice tasks, although with- out having the diagram available and without having the chance of retries until the correct response was found. This time the experimental group had better control performance (Preußler, 1998). These findings have been interpreted as demonstrating that structural knowledge needs to be either actively acquired or prac- ticed in the context of application in order to benefit control performance (Preußler, 1996, 1998; Schoppek, 2004), a notion that resonates with the broader liter- ature on "learning by doing" and cognitive skill acqui- sition (e.g. Anderson, 1993). An alternative explanation is that the "guidance" given to participants in these studies was not suf- ficient to immediately promote structural knowledge (Goode & Beckmann, 2010). Specifically, Goode and Beckmann (2010) argue that in Putz-Osterloh’s (1993) study participants may not have understood how the diagram related to changing the input and output vari- ables. In order to understand the meaning of the di- agram, problem solvers may require a direct demon- stration of how the inputs affect the output. Whilst problem solvers in Preußler’s, (1996) study did receive an explanation as to how the inputs affect the outputs, they did not receive a structural diagram. It is likely that they may have been unable to recall this informa- tion during the control task. Goode and Beckmann (2010) developed instructional material to overcome these limitations, and compared control performance under conditions of complete, partial or no structural information. They used a CPS task with three in- puts and three outputs, interconnected by six linear relations. As in Putz-Osterloh’s (1993) and Preußler’s (1996, 1998) studies, the labels given to the system variables did not refer to objects in the real world (e.g. "A", "B", "C"). The instructional material included an audio-visual demonstration of how the inputs affect the outputs, and the formation of a causal diagram as a result of the interventions shown. This information was then available on screen during the subsequent control phase. They found that problem solvers who received complete information were significantly bet- ter at controlling the system than those who received partial or no information. This study illustrates that structural information can have an immediate positive impact on the quality of control performance, without a period of goal-orientated practice or prior exposure to the system. The findings suggest that the effective- ness of providing structural information is more a mat- ter of accessibility (i.e., instructional design), rather than practice. Another factor that appears to influence the ap- plication of structural information is the complexity of the underlying structure of the CPS task. In a follow-up study to Goode and Beckmann (2010) using the same methodology, Goode (2011) provided partic- ipants with either complete, partial or no structural information regarding the underlying structure of one of four CPS tasks, which varied in system complex- ity. The complexity of the tasks was manipulated by increasing the number of relations that had to be pro- cessed in parallel in order to make a decision about a particular goal state (i.e., the connectivity of the goal state). The study showed that it was more difficult for subjects to understand and utilise information as system complexity increased; floor effects on perfor- mance were observed when three relations had to be considered in parallel to make a decision about a goal state. This may also explain why previous studies have found that structural information did not benefit con- trol performance (Putz-Osterloh, 1993; Preußler, 1996, as "LINAS" is at this level of complexity. Süß’s (1996) study employed a task with many more variables and relations. This may have made it more difficult for subjects to understand and utilise the information that they were given. Nevertheless, the issue of whether the provision of structural information results in immediate improve- ments for control performance after knowledge has al- ready been acquired through an unguided exploration remains unresolved. Goode and Beckmann’s (2010) 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 3 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems and Goode’s (2011) studies did not include a compar- ison with a group who acquired knowledge through an unguided exploration of the system variables. The aim of the current study is to determine whether structural information can directly benefit control performance. To allow direct comparisons to control performance scores from Goode and Beckmann (2010), this study will use the same CPS task, inter- vention and performance goals. In our proposed de- sign, participants will first explore a CPS task, and try to independently acquire knowledge about its un- derlying structure. They will then try to control the system to reach specific goal values of the output vari- ables. Participants in an experimental condition will then watch an audio-visual demonstration of how the inputs affect the outputs, and will observe the forma- tion of a causal diagram as a result of the interven- tions shown (as per the procedure reported in Goode and Beckmann, 2010 and Goode, 2011). Both the ex- perimental and the control group will then control the system again. If structural information can be im- mediately utilised, then problem solvers who receive structural information should show an improvement in their control performance, and should be better con- trolling the system than those who have to rely on the knowledge they acquired independently (see Informa- tion Hypothesis). The role of fluid intelligence in benefiting from structural information A second issue addressed by this study is whether ben- efiting from structural information is dependent on the cognitive abilities of the problem solver. Gold- man (2009) has argued that learner characteristics, such as prior knowledge and cognitive ability, deter- mine whether benefits are derived from instructional settings. The CPS task employed in the current study uses abstract variable labels and a domain-neutral cover story. This aims at minimising confounding ef- fects of individual differences in domain-specific knowl- edge on the results (for a detailed discussion of the ar- gument for using abstract systems in complex problem solving research see Beckmann & Goode, 2014). Con- sequently, the guidance information provided (i.e., the intervention) is expected to be relatively novel for all participants, so that individual differences in utilising it can largely be attributed to the cognitive abilities of the problem solver. Previous findings show that when explicit informa- tion about system structure is provided, control per- formance is consistently moderately to strongly cor- related with fluid intelligence (Bühner, Kröner, & Ziegler, 2008; Goode & Beckmann, 2010; Kröner et al., 2005; Putz-Osterloh, 1981; Putz-Osterloh & Lüer, 1981; Wüstenberg et al., 2012). Therefore, in the current study it is predicted that under conditions where participants receive information, the extent of improvements in control performance will be a func- tion of their fluid intelligence as measured via APM. In comparison, the extent of improvements in control performance when participants do not receive addi- tional information should be due to practice applying their partial representations of the underlying struc- ture, and therefore less strongly related to fluid intelli- gence as measured via APM (see Intelligence Hypoth- esis). Aims and hypotheses In summary, the main goal of this paper is to deter- mine whether guidance in the form of structural infor- mation results in an immediate improvement in con- trolling a CPS task after knowledge has already been acquired through an unguided exploration of the sys- tem variables. A secondary aim is to examine whether any improvements are moderated by fluid intelligence as measured via APM. Firstly, it is hypothesised that participants who acquire more knowledge during the exploration phase about the underlying structure of the system should show better control performance prior to any instructional intervention (Knowledge Hy- pothesis). Secondly, participants who receive struc- tural information should improve their control perfor- mance more than those who receive no additional in- formation (Information Hypothesis). Thirdly, under conditions where participants receive information, the magnitude of this improvement will be a function of their fluid intelligence as measured via APM (see In- telligence Hypothesis). Method Participants Ninety-eight first year psychology students at the Uni- versity of Sydney, Australia, participated for course credit. Nine participants failed to complete all tasks therefore their data were excluded from further anal- ysis. The available sample size of about 100 would guarantee sufficient statistical power (1 − β ≥ .80) in identifying at least medium effects (d = 0.50) at a sig- nificance level of α ≤ .05 (one-tailed) in the planned analyses. Design Participants were randomly assigned to one of two conditions (45 participants in the Information condi- tion, 44 participants in No Information condition). As problem solvers were required to control the system on two occasions this resulted in a 2 x 2 design. The within-subjects factor was control performance (cy- cle 1 and cycle 2). The between-subjects factor was whether or not they received structural information (Information and No Information). The aim of the intervention in the information condition was to en- courage participants to develop a complete and accu- rate representation of the underlying structure of the system. The no information condition represented a passive control group. Participants were assessed on 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 4 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems their structural knowledge, control performance for cy- cle 1, control performance for cycle 2 and performance in a test of fluid intelligence. Vary-one-thing-at-time (VOTAT) strategy use during the exploration phase was also assessed as part of this study; this measure is not reported in this paper. Figure 1 displays the procedure of the experiment for each condition and indicates which performance measures were collected in each phase of the experiment. Figure 1. Diagram for the procedure of the experiment, illustrating the phases of the experiment by condition and indicating which performance measures were collected in each phase. Description of CPS task The CPS task was programmed using Adobe Flash 8 and Captivate 3, and administered on PCs (see (Goode, 2011 for an extensive description of all of the CPS task elements, including step-by-step screenshots of the instructional intervention and CPS task, and transcript of the explanation). The underlying structure was originally developed by Beckmann (1994, see also Beckmann & Goode, 2014; Goode & Beckmann, 2010; Goode, 2011), and is based on the approach to complex problem solving that was developed by Funke (1992) in his DYNAMIS research project. It consists of three input and three output variables that are connected by a set of linear equations: Xt+1 := 1.0 ∗ Xt + 0.8 ∗ At + 0.8 ∗ Bt + 0.0 ∗ Ct Yt+1 := 0.8 ∗ Yt + 1.6 ∗ At + 0.0 ∗ Bt + 0.0 ∗ Ct Zt+1 := 1.2 ∗ Zt + 0.0 ∗ At + 0.0 ∗ Bt + a.0 ∗ Ct Xt,Yt and Zt denote the values of the output vari- ables and At,Bt and Ctdenote the values of the input variables during the present trial whilst Xt+1,Yt+1 and Zt+1 denote the values of the output variables in the subsequent trial. Important for the operationalization of knowledge acquisition, the system can be considered balanced, i.e., from 12 possible relationships between variables 6 do exist and among the three output variables one is subject to a "positive" eigendynamic (i.e., an au- toregressive dependency that results in a monotone increase), one is subject to a negative eigendynamic (i.e., an autoregressive dependency that results in a monotonic decrease) and one is subjected to no eigen- dynamic and all three output variables have a double dependency. Previous research has found that the presence of a semantically meaningful context has an unpredictable, often negative, effect on acquisition of structural knowledge (Beckmann, 1994, see also Beckmann & Goode, 2014; Burns & Vollmeyer, 2002; Lazonder, Wilhelm, & Hagemans, 2008; Lazonder, Wilhelm & van Lieburg, 2009.) Therefore, in order to ensure that the system was relatively novel for all participants and thus control the potential influence of prior knowledge, the input and output variables are labelled with let- ters. As can be seen in Figure 2, the output variables are labelled X, Y and Z, whilst the input variables are labelled A, B, and C. The user-interface is in a non-numerical graphical format, in order to encourage the formation of mental representations more aligned with the development of causal diagrams. In accordance with the principles of cognitive load theory (CLT), this should minimise the cognitive activities that are not directly relevant to the task (extraneous cognitive load, e.g. Sweller, 1994; for a review see Beckmann, 2010; Sweller, 2010). Figure 2 shows that the values of the input variables are represented as bars of varying heights in the boxes on the input variables, where positive values are shown above the input line and negative values are shown below. Each box represents the value of the input variable on a single trial, and in total seven trials can be conducted before the values are reset (representing a single cycle). Although the numerical values of the inputs are not available to participants, the inputs are varied in increments of one unit, within the range of -10 to 10. On each trial, participants have to set value of each input variable. This is done step by step, such that after they set the value for input A, they then have to set the value for input B and finally input C before the resulting values of the output variables are displayed as line graphs below. Previous inputs and there subse- quent effects remained on the screen (decision history) for each cycle of seven trials. There was no time limit. During the control phase of the task, the goals are indicated as dotted lines on the graphs for the output variables (as show in Figure 2). The target values used for control cycle 1 and 2 were of comparable in diffi- culty, i.e., the Euclidean Distance (see operationaliza- tion of control performance) between start values and target values was the same for both cycles. Dependent variables and individual differences measure Structural knowledge Participants’ structural knowledge was assessed by asking them to create causal diagrams of the relation- ships between the input and output variables at the end of each trial during the exploration phase. The 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 5 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems Figure 2. Screenshot of the system interface, as presented in the information condition after the instructional phase. The goals are indicated as dotted lines on the graphs for the output variables. The underlying structure of the system is represented on screen as a causal diagram, where the arrows represent the relationships be- tween the variables, while the positive and negative signs denote the direction of the relationship, and the letters the relative strength. In this example, for the fifth trial of seven all input variables were increased (A half the strength and B and C using the maximum); as a result, Output X and Y increased whilst Output Y decreased slightly. diagram that was generated on the final exploration trial (after 2 cycles of 7 trials), before the control phase was used to derive a structural knowledge score. Us- ing a procedure introduced by Beckmann (1994), the operationalisation of the knowledge acquisition perfor- mance is based on a threshold model for signal de- tection (Snodgrass & Corwin, 1988). The proportion of correctly identified relationships was adjusted for guessing by subtracting the proportion of incorrectly identified relationships. The final score has a theoret- ical range from -.98 to .98, where a score below zero indicates inaccurate knowledge, whilst a score above zero indicates more accurate knowledge. Control performance The scoring procedure used was based on Beckmann’s (1994) scoring system. Control performance was calcu- lated by determining the Euclidean Distance between the vectors of actual and optimal values of the input variables. The ideal values for each input variable, i.e., the intervention that would result in the system reach- ing the goal state, were calculated by using the values of the output variables on the previous trial and the goal output values to solve the set of linear equations underlying the system. As the range of possible input values is restricted for the system used in this study (i.e., between -10 and 10), it might not be possible to bring the system into the goal state by a single inter- vention. In cases as these, i.e., when the ideal values fall outside this range, the values were adjusted to the nearest possible values, which then constituted the op- timal values. In cases when the ideal values are within the range of possible inputs, the ideal values were used as the optimal input. For the system at hand the theoretical range of this score is 0 to 34, where a lower score indicated a smaller deviation from optimal control interventions and therefore better performance. Fluid intelligence The percentage of correct responses on an abridged version of the Raven’s APM (Raven et al., 1998) was used as an indicator of fluid intelligence. This version of the APM included 20 items from the original 36- item test, created using the odd numbered items plus 2 additional even numbered ones from the most complex items (i.e. items 34 and 36). Procedure The CPS task and the APM were presented to partic- ipants on PCs, over two separate sessions. The CPS task for each condition was installed on alternate com- puters at the study venue. On arrival at the first ses- sion, participants chose a computer, which determined their condition. The CPS task began with a set of instructions that explained the user-interface, how to change the values of the input variables and how to record and alter the causal diagram. At the end of the instructions, par- ticipants were informed that the task consists of two phases. Firstly, they had to explore the system to dis- cover the underlying structure of the system and then control the system to reach certain values of the out- put variables. The goal values were not revealed until the beginning of each control cycle. The exploration phase then began, in which par- ticipants were prompted to explore the system for two cycles of 7 trials each by changing any of the input vari- ables and observing the effect on the output variables displayed in the graphs. At the end of each trial, par- ticipants had to record what they had learned about the system using the causal diagram construction fea- ture that was displayed on the screen. The causal diagram could be altered using a set of twelve buttons (one for each possible relationship in the system) at the bottom of the screen. Each but- ton referred to a particular relationship in the system. Using these buttons, participants could record if they thought there was a relationship between two variables or not, or if they thought the output variables changed independently (or not). They could also specify the di- rection of the effect, and its perceived strength. After the exploration phase, participants then had to control the system by manipulating the inputs to reach set values of the outputs for seven trials, which were indicated as dotted lines on the output graphs (Control Cycle 1). The causal diagrams they had constructed during the exploration phase remained on screen, providing access to the structural information they had individually extracted. In the information condition, participants then watched an instructional video that explained the ac- tual underlying structure of the system. The instruc- 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 6 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems tions were designed in accordance with the principles of CLT, and the aim was to reduce the amount of cog- nitive activities that problem solvers would have to undertake to translate the information provided into knowledge about the system (minimising extraneous cognitive load). In particular, previous research has shown that learning is facilitated when explanations of graphical information is presented aurally, rather than as text (modality effect, Tabbers, Martens, & van Merriënboer, 2004). Therefore, the instructions con- sisted of a recording of seven intervention trials with an accompanying audio narration, which explained the actual underlying structure of the system. After each trial, the narrator explained how each of the outputs had changed, and how this reflected the underlying structure of the system. The respective causal dia- gram was constructed on screen in parallel, to record this information. Participants in the no information condition did not receive any additional information during this phase; representing a passive control group. All participants then had to control the system again for seven trials, with different goals indicated on the output variables (Control Cycle 2). In the informa- tion condition the causal diagram displayed onscreen was the correct and complete one. In the no informa- tion condition, the causal diagram that participants had constructed in the initial exploration cycles was displayed onscreen. In a subsequent session, approximately one week later, participants completed the APM. Data Analysis To test our main hypotheses we conducted a series of hierarchical linear modelling analyses using the HLM software package (Raudenbush, Bryk, Cheong, & Con- gdon, 2000). This approach allows us to model indi- viduals’ change in performance from control cycle 1 to control cycle 2 as function of person-level variables (see Raudenbush & Bryk, 2002)). We used a two level model in which performance in control cycle 1 and con- trol cycle 2 (level 1) were clustered within individuals (level 2). The specific analyses that we performed to test each hypothesis are discussed in the results sec- tion. Results The following sections first present preliminary analy- ses undertaken to test whether the random assignment to condition was effective, and justify our treatment of the variables in the following analyses. The findings in relation to the three hypotheses are then presented. Intercorrelations (Pearson) between the variables used in this study as well as descriptive statistics and distributions are presented in Table 1. The distribu- tions of the variables indicate that assumptions of nor- mality were met. Equivalence between the conditions prior to the intervention To examine the effect of the intervention and fluid in- telligence on control performance, firstly, it was nec- essary to check whether the conditions differed prior to the intervention. The amount of structural knowl- edge acquired by participants during the exploration phase did not differ by condition; t(87) = -.09, p = .93, d = 0.02, nor did their control performance scores in cycle 1; t(87) = -.05, p = .96, d = 0.01, or scores on the APM; t(87) = 1.59, p = .12, d = 0.34. This suggests that the procedure used to randomly allocate participants to the conditions was effective. Structural knowledge acquired during the exploration phase For both conditions, the amount of structural knowl- edge that was acquired during the exploration phase was significantly greater than zero; M = .22, SD = .34, t(88) = 6.00, p < .01. This indicates that on average, participants had acquired some knowledge of the un- derlying structure of the system prior to the first con- trol cycle. However, the range of structural knowledge scores, -.49 to .98, indicates that participants differed widely in the amount of knowledge that they were able to acquire about the underlying structure of the system during the initial exploration phase. That is, while some participants were able to acquire complete knowledge of the underlying structure of the system (one participant in the no information condition, and two participants in the information condition), oth- ers acquired a rather incorrect representation of the underlying structure. This also suggests that for the majority of problem solvers the provision of structural information could potentially represent a significant source of new information about the underlying struc- ture of the system. Internal consistencies Internal consistency analyses were conducted to de- termine the variability in control performance scores across the trials and for different goal states as an esti- mate of the reliability of the dependent variables. In- ternal consistency was good across the first control cy- cle (ω = .85, 95% CI [.79, .89]) and the second control cycle (ω =.93, 95% CI [.89, .95]) (Dunn, Baguley, & Brunsden, 2014). This indicates that problem solvers are rather consistent in their performance and it jus- tifies averaging the scores across each control cycle. A further analysis indicated that the reliability of the APM scores was acceptable across the 20 items (ω = .76, 95% CI [.59, .83]) (Dunn et al., 2014). Knowledge Hypothesis In support of the knowledge hypothesis, across the conditions, there was a significant moderate negative relationship between structural knowledge scores and control performance in cycle 1 (r = -.34, p < .01). 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 7 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems Table 1. Descriptive statistics, distributions and inter-correlations (Pearson) between the variables for each condition. M Min. Max. Kurtosis Skewness 2 3 4 (SD) (SE) (SE) No Information Condition 1. Structural Knowledge .21 -.49 .98 -.40 .38 -.38* -.51** .45** (.33) (.70) (.36) N = 44 2. Cycle 1 13.88 4.52 20.77 -.54 -.58 . . . .56** -.24 (4.20) (.70) (.36) 3. Cycle 2 13.21 2.50 25.01 -.58 -.31 . . . . . . -.18 (5.28) (.70) (.36) 4. APM 63.75 30 95 -.23 -.07 . . . . . . . . . (16.32) (.70) (.36) Information Condition 1. Structural Knowledge .22 -.33 .98 -.47 .29 -.31* -.36* .26 (.35) (.69) (.35) N = 45 2. Cycle 1 13.92 2.82 24.09 .75 -.49 . . . .27 -.11 (4.14) (.69) (.35) 3. Cycle 2 10.24 2.10 22.33 -1.03 .35 . . . . . . -.52** (5.29) (.70) (.35) 4. APM 58.00 30 95 -.85 .06 . . . . . . . . . (17.75) (.69) (.35) Note. ∗ p < .05. ∗∗ p < .01. This indicates that participants who acquired more knowledge about the underlying structure of the task produced smaller deviations from the set of optimal control interventions, and were therefore better at controlling the system (i.e., reaching and maintain the goal values). This advantage persisted in cycle 2 even when participants received additional instruc- tions with regard to the underlying structure of the system (r information = -.36, p < .01, r no information = -.51, p < .01). Information and Intelligence Hypotheses In order to determine whether the provision of struc- tural information facilitates control performance (In- formation Hypothesis) and whether the extraction of knowledge from information in this context is de- termined by fluid intelligence (Intelligence Hypothe- sis) we conducted a series of two-level HLM analy- ses. Firstly, a random coefficient regression analy- sis was conducted to assess whether control perfor- mance changed across the two control cycles. At level 1, each participant’s performance was represented by an intercept term, which denoted their mean perfor- mance across control cycle 1 and control cycle 2, and a slope, that represented their change in performance from control cycle 1 to 2. Control cycle (1 or 2, effect coded as -.5 and .5, respectively) was entered as an independent variable at this level. The mean control performance scores and the change in control perfor- mance then became the outcome variables in a level-2 model, in which they were modelled as random ef- fects. The results of this analysis are presented in the top section of Table 2. This analysis indicated that the mean control performance score was 12.81 across control cycle 1 and 2 and on average, control performance scores improved by 2.19 points from con- trol cycle 1 to 2. The change in control performance was significantly different from zero; t(88) = -3.86, p < .001. There was significant differences between problem solvers in terms of their mean control perfor- mance scores and the change in their control perfor- mance; χ2 = 2867895259.6, df = 88, p < .001 and χ2 = 1274245149.4, df = 88, p < .001, respectively. Variability in problem solvers’ change in control per- formance from control cycle 1 to 2 accounted for 64% of the total variability in control performance scores. These findings are an important prerequisite for the subsequent analyses, as they indicate that individu- als show substantial variability in their mean control performance and the extent to which their control per- formance changed across the two cycles. We conducted an intercept- and slope-as-outcomes regression analysis in which mean control performance and the change in control performance from control cy- cle 1 to 2 were modelled as a function of condition (as an effect coded variable indicating condition: -.5 = no information, .5 = information) and scores on the APM at level 2. The level 1 model was the same as in the random coefficients regression analysis. The results of this analysis are presented in the middle panel of Table 2. With regard to the Information Hypothesis, this analysis indicated that information had a significant impact on average control performance scores, and the change in control performance from control cycle 1 to 2, controlling for the effects of fluid intelligence; t(86) = -2.32, p < .05, ∆R2 = 5% and t(86) = -3.19, p < .01, ∆R2 = 9%, respectively. Participants in the information condition had an average control perfor- mance score 1.91 points better than those in the no information condition. Similarly, the change in control performance for participants in the information condi- tion was 3.42 points better than those in the no infor- mation condition. In support of the Information Hy- pothesis, these results indicate that participants who received additional information with regard to the un- derlying structure of the system performed better on average, and improved at a greater rate from control cycle 1 to control cycle 2 than those who did not re- ceive information. 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 8 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems With regard to the Intelligence Hypothesis, the analysis also indicated that APM scores were signif- icantly linked to control performance scores as well as to their change from control cycle 1 to 2, controlling for the effects of condition; t(86) = -3.21, p < .01, ∆R2 = 7% and t(86) = -2.17, p < .05, ∆R2 = 2%, re- spectively. On average, a one-point increase in scores on the APM was associated with a 0.08 better score on average control performance, and a 0.07 better score on the change in performance scores from control cycle 1 to 2. These results indicate that on average, partic- ipants with a higher APM scores, tended to perform better overall, and improved more from control cycle 1 to 2. In order to determine whether the effect of fluid in- telligence on control performance differed by condition a third analysis was conducted in which an interaction term (APM x Condition) was added to the main effects of the variables at level 2. The results are presented in the bottom panel of Table 2. There was no evidence that fluid intelligence (as measured via APM scores) has an effect on mean control performance scores var- ied by condition, as the interaction term was small and insignificant; t(85) = -.68, p = .50, ∆R2 = 0%. However, the effect of fluid intelligence on the change in performance from control cycle 1 to control cycle 2 did vary significantly by condition; t(85) = -2.48, p < .05, ∆R2 = 3%. In further support of the In- telligence Hypothesis, this suggests that the change in performance scores for participants who received infor- mation was more strongly related to fluid intelligence than for participants who did not receive information. Discussion This study examined whether: (1) providing guidance in the form of structural information results in an im- mediate improvement in controlling a CPS task after knowledge has already been acquired through an un- guided exploration of the system variables; and (2) any improvements are moderated by fluid intelligence as measured via APM. In summary, support was found for the Knowledge Hypothesis, as participants who ac- quired more structural knowledge during the explo- ration phase had better control performance in con- trol cycle 1. Support was also found for the Informa- tion Hypothesis, as participants who received struc- tural information improved their control performance more than those who received no information. Finally, support was found for the Intelligence Hypothesis, as when participants received information, their change in control performance scores from control cycle 1 to 2 was more strongly related to APM performance scores than the change in control performance scores in par- ticipants who did not receive structural information. These results suggest that guidance in the form of structural information does confer an additional ad- vantage in controlling a complex system over indepen- dently acquired knowledge, and that problem solvers can translate such information into effective control ac- tions without practice. However, the extent to which problem solvers can benefit from such information ap- pears to be moderated by their fluid intelligence as measured via APM. As in previous studies, it was found that the amount of structural knowledge acquired by participants is strongly related to the quality of their control per- formance. In addition, in line with other studies, few participants were able to acquire complete knowl- edge of the underlying structure of the system during the exploration phase (Beckmann, 1994; Beckmann & Guthke, 1995; Burns & Vollmeyer, 2002; Funke & Müller, 1988; Müller, 1993; Kröner, 2001; Kröner et al., 2005; Kluge, 2008; Osman, 2008; Schoppek, 2002; Vollmeyer et al., 1996). These findings provide fur- ther evidence that learners require additional support or guidance to acquire complete and accurate knowl- edge about complex and dynamic systems; they are unlikely to do so through unguided discovery learning. This study also found that guidance in form of pro- viding structural information resulted in an immediate improvement in control performance. In contrast to previous studies (Preußler, 1998; Putz-Osterloh, 1993; Süß, 1996), these findings suggest that a period of ac- tive practice is not required to translate knowledge into effective control actions. One caveat to this con- clusion is, however, that the task used in other studies could be considered more complex than the task used in the current study. Further studies are required to determine whether the findings observed in this study generalise to more complex tasks. Nevertheless, the results support and extend upon the findings of Goode and Beckmann (2010) in impor- tant ways. As in Goode and Beckmann’s (2010) study, the results of the present study show that if problem solvers receive a direct demonstration as to how each input affects each output, and have access to this in- formation in form of a causal diagram during control performance, then they will be able to immediately translate this information into the appropriate actions for controlling the system. This provides further sup- port for the claim that supporting information should be available throughout the task (Berry & Broadbent, 1987; Gardner & Berry, 1995; Leutner, 1993). Indeed, comparing the findings from the current study with Goode and Beckmann’s (2010) study, which employed the same CPS task, instructional method and participants drawn from the same uni- versity student population, suggests that an unguided, albeit "active" exploration of the system variables pro- vides no advantage for control performance whatso- ever. In Goode and Beckmann’s (2010) study, par- ticipants received structural information and then were required to immediately control the system vari- ables; mean control performance scores were 10.33 (SD = 5.25) in the comparable control cycle. In the current study, mean control performance scores were 10.24 (SD = 5.29). This suggests that the actively acquired knowledge and practice controlling the sys- tem variables resulted in no net advantage for control performance over simply providing structural informa- 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 9 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems Table 2. Results of the Random Coefficients Regression (RCR) Analysis and the Intercept- and Slope-As-Outcome Regression (ISAOR) Analyses Variable Parameter Estimate SE t ∆R2 RCR Analysis Mean control performance (β00) 12.81 0.43 30.10** Mean change in control performance (β10) -2.19 0.57 -3.86** ISAOR Analysis 1 Intercept-as-outcome Condition (β01) -1.91 0.82 -2.32* 5% APM (β02) -0.08 0.02 -3.21** 7% Slope-as-outcome Condition (β11) -3.42 1.07 -3.19** 9% APM (β12) -.07 0.03 -2.17* 2% ISAOR Analysis 2 Intercept-as-outcome Condition (β01) -1.89 .82 -2.33* 5% APM (β02) -0.08 .02 -3.32** 7% Condition x APM (β03) -0.03 .04 -0.68 0% Slope-as-outcome Condition (β11) -3.38 1.03 -3.29** 9% APM (β12) -0.06 .03 -2.37* 2% Condition x APM (β13) -0.13 0.05 -2.48* 3% Note. ∗ p < .05. ∗∗ p < .01. Level 1 model (for all analyses): Yti = π0i + πli(Control Cycle), where Yti is person i’s control performance score at time t, π0i is their mean control performance score and πli is their change in control performance from control cycle 1 to control cycle 2. Level 2 model for RCR Analysis: π0i = β00 + r0i and πli = β10 + rli Level 2 model for ISAOR Analysis 1: π0i = β00 + β01(Condition) + β02(APM) + r0i and πli = β10 + β11(Condition) + β12(APM) + rli Level 2 model for ISAOR Analysis 2: π0i = β00 + β01(Condition) + β02(APM) + β03(Condition x APM) + r0i and πli = β10 + β11(Condition) + β12(APM) + β13(Condition x APM) + rli Note: When intercepts are outcomes, ∆R2 is expressed as a percentage of the variability in mean control performance scores. When slopes are outcomes, ∆R2 is expressed as a percentage of the variability in the change in control performance. 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 10 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems tion. The finding that participants in the no informa- tion condition showed little improvement across the control cycles further reinforces this claim. This sug- gests that practice at controlling the system does not have a significant impact upon the quality of problem solvers’ control performance, especially if the control goals change. Indeed, under both conditions the high level of in- ternal consistency in control performance scores fur- ther suggests that problem solvers do not dramatically change their control behaviours through practice. Sub- sequently, improvements in control performance with practice are rather limited. In other words, these re- sults seem to suggest that no spontaneous optimisa- tion of control behaviour (i.e. learning by doing) takes place. The question, however, of whether longer peri- ods of active practice after exposure to guiding infor- mation, would lead to further improvements, could be of interest in future studies. These findings are consistent with recent findings regarding CPS training. Kretzschmar and Süß (2015) trained participants using five different computer- based complex dynamic systems, and their perfor- mance was tested in a sixth system. Interacting with each system involved a goal-free exploration phase and a control phase. They found that trained participants were able to acquire more knowledge about the final system than an untrained control group. However, there was no difference in control performance. In line with the findings from our study, this suggests that for each control intervention, the problem solver must apply their knowledge to generate the correct action for that specific situation. With regard to the relationship between fluid in- telligence and control performance, it should first be acknowledged that the generalisability of the results from this study may be limited by the narrow opera- tionalisation of fluid intelligence via APM. Whilst the APM has been traditionally seen as the empirical refer- ence point of fluid intelligence, more recent discussions (e.g., Gignac, 2015) are critical of studies that rely on this single test score. This on-going debate should be kept in mind while reading the following interpretation of the findings. The results of this study are in line with previous studies that have shown that when structural infor- mation is provided, control performance is moderately to strongly correlated with fluid intelligence (Bühner et al., 2008; Goode & Beckmann, 2010; Kröner et al., 2005; Putz-Osterloh, 1981; Putz-Osterloh & Lüer, 1981; Wüstenberg et al., 2012). This suggests that more intellectually capable problem solvers are able to make use of structural information more effectively than individuals who are less so. This study extends on these previous findings, as it was also found that fluid intelligence as measured via APM had an impact on the acquisition of structural knowledge during the exploration phase, and subsequently in controlling the system when only incomplete knowledge was available. These results suggest that intellectually more capable problem solvers are at a double advantage in compari- son to those who score lower on fluid intelligence with regard to acquiring and utilising structural knowledge: they are able to acquire more knowledge without as- sistance, and they also benefit more from guidance. This implies a necessity to tailor instructions to prob- lem solvers’ intellectual capacity, an aspect often ne- glected in educational contexts. In other words, and as frequently advanced by Snow (1986; 1989; Snow & Lohman, 1989; Snow & Yallow, 1982), individual differences among learners still "...present a pervasive and profound problem to educators" (Snow, 1989, p. 1029). The results with regard to the role of fluid intelli- gence also provide support for the claim that in pre- vious studies (Preußler, 1996; Putz-Osterloh & Lüer, 1981; Putz-Osterloh, 1993), the effect of structural information on control performance may have been masked by individual differences in the ability to un- derstand and utilise the information. In addition, Preußler’s (1998) finding that all of her participants were able to effectively utilise information after a pe- riod of active practice, may now be interpreted in a different light. It may be that practice per se is not the essential component, but rather that some prob- lem solvers require more extensive guidance in order to be able to make sense of the information that is provided. Overall, our results imply that guidance in the form of structural information has the potential to provide benefits over and above the effects of discovery learn- ing. The crucial aspect of guidance, however, is that it is well designed. These findings are in line with those from other domains that show that learners experience many difficulties when they are required to indepen- dently acquire knowledge without guidance (de Jong & van Joolingen, 1998; Mayer, 2004; de Jong, 2005; 2006; Kirschner et al., 2006). Acknowledgements: This manuscript is based on a chapter from a doctoral thesis (Study 1; Goode, 2011). The research was undertaken at the University of Syd- ney and funded through an Australian Postgraduate Award. This research was also partly supported under Australian Research Council’s Linkage Project funding scheme (project LP0669552). Declaration of conflicting interests: The authors de- clare that the research was conducted in the absence of any commercial or financial relationships that could be constructed as a potential conflict of interest. Handling editor: Andreas Fischer Author contributions: The authors contributed equally to this work. Supplementary material: Supplementary material available online. 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 11 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems Copyright: This work is licensed under a Creative Com- mons Attribution-NonCommercial-NoDerivatives 4.0 In- ternational License. Citation: Goode, N. & Beckmann, J. F. (2016). With a little help. . . : On the role of guidance in the acquisition and utilisation of knowledge in the control of complex, dynamic systems. Journal of Dynamic Decision Making, 2, 5. doi:10.11588/jddm.2016.1.33346 Received: 19 September 2016 Accepted: 01 January 2017 Published: 26 February 2017 References Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learn- ing? Journal of Educational Psychology, 103(1), 1–18. doi:10.1037/a0021017 Anderson, J.R. (1993). Rules of the mind. Hillsdale, NJ: Erlbaum. doi:10.4324/9781315806938 Beckmann, J. F. (1994). Lernen und komplexes Problemlösen: Ein Beitrag zur Konstruktvalidierung von Lerntests [Learning and complex problem solving: A contribution to the construct valida- tion of tests of learning potential]. Bonn, Germany: Holos. Beckmann, J. F. & Goode, N. (2014). The benefit of being naïve and knowing it: the unfavourable impact of perceived context familiarity on learning in complex problem solving tasks. Instruc- tional Science, 42(2), 271-290. doi:10.1007/s11251-013-9280-7 Beckmann, J.F. & Guthke, J. (1995). Complex problem solving, in- telligence and learning ability. In P.A. Frensch & J. Funke (Eds.) Complex Problem Solving: The European Perspective (pp. 177–200). Hillsdale, NJ: Erlbaum. doi:10.4324/9781315806723 Beckmann, J.F. (2010). Taming a beast of burden – On some issues with the conceptualisation and operationalisation of cognitive load. Learning and Instruction, 20, 250-264. doi:10.1016/j.learninstruc.2009.02.024 Berry, D.C. & Broadbent, D.E. (1984) On the relationship between task performance and associated verbalizable knowledge. The Quarterly Journal of Experimental Psychology Section A, 36(2), 209-231. doi:10.1080/14640748408402156 Berry, D.C. & Broadbent, D.E. (1987) Explanation and verbaliza- tion in a computer-assisted search task. The Quarterly Jour- nal of Experimental Psychology Section A, 39(4), 585-609. doi:10.1080/14640748708401804 Blech, C. & Funke, J. (2005). Dynamis review: An overview about applications of the Dynamis approach in cognitive psy- chology. Heidelberg University: Unpublished manuscript. Re- trieved from: https://www.psychologie.uni-heidelberg.de/ae/allg _en/forschun/dynamis/dynamis_review_08-2005.pdf Bühner, M., Kröner, S., & Ziegler, M. (2008). Work- ing memory, visual-spatial intelligence and their rela- tionship to problem-solving. Intelligence, 36, 672–680. doi:10.1016/j.intell.2008.03.008 Burns, B. & Vollmeyer, R. (2002). Goal specificity effects on hypothesis testing in problem solving. The Quarterly Jour- nal of Experimental Psychology Section A, 55(1), 241-261. doi:10.1080/02724980143000262 de Jong, T. & van Joolingen W.R. (1997). An extended dual search space model of scientific discovery learning. Instructional Science, 25, 307-346. doi:10.1023/A:1002993406499 de Jong, T. & van Joolingen W.R. (1998) Scientific discovery learn- ing with computer simulations of conceptual domains. Review of Educational Research, 68, 179–202. doi:10.2307/1170753 de Jong, T., Linn, M. C., & Zacharia, Z. C. (2013). Physical and virtual laboratories in science and engineering education. Science, 340, 305–308. doi:10.1126/science.1230579 de Jong, T. (2005). The guided discovery principle in multimedia learning. In R.E. Mayer (Ed.), Cambridge handbook of mul- timedia learning (pp. 215–229). Cambridge, UK: Cambridge University Press. doi:10.1017/cbo9780511816819 de Jong, T. (2006). Scaffolds for Computer Simulating Based Sci- entific Discovery Learning. In J. Elen & R. E. Clark (Eds.), Han- dling Complexity in Learning Environments: Theory and Research (Advances in Learning and Instruction) (pp. 107-128). Oxford, UK; Boston: Elsevier. Danner, D., Hagemann, D., Holt, D.V., Hager, M., Schankin, A., Wüstenberg, S., & Funke, J. (2011). Measuring Performance in Dynamic Decision Making: Reliability and Validity of the Tailor- shop Simulation. Journal of Individual Differences, 32, 225-233. doi:10.1027/1614-0001/a000055 Dörner, D. (1980). On the difficulties people have in deal- ing with complexity. Simulation and Games, 11, 87-106. doi:10.1177/104687818001100108 Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399-412. doi:10.1111/bjop.12046 Funke, J. & Müller, H. (1988). Eingreifen und Prognostizieren als Determinanten von Systemidentifikation und Systemsteuerung [Intervention and prediction as determinants of system identi- fication and system control]. Sprache & Kognition, 7, 176-186. Funke, J. (1992). Dealing with dynamic systems: Research strat- egy, diagnostic approach and experimental results. German Journal of Psychology, 16(1), 24-43. Retrieved from http:// cogprints.org/3004/1/Funke_1992_GJP.pdf Funke, J. (2001). Dynamic systems as tools for analysing human judgment. Thinking and Reasoning, 7, 69–89. doi:10.1080/13546780042000046 Gardner, P. H. & Berry, D. C. (1995). The effect of different forms of advice on the control of a simulated complex system. Applied Cognitive Psychology, 9(7). doi: doi:10.1002/acp.2350090706 Gignac, G. E. (2015). Raven’s is not a pure measure of general intelligence: Implications for g factor theory and the brief measurement of g. Intelligence, 52, 71- 79. doi:10.1016/j.intell.2015.07.006 Gobert, J.D. & Clement, J.J. (1999). Effects of student-generated diagrams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science Teaching, 36(1), 39-53. doi:10.1002/(SICI)1098-2736(199901)36:1<39::AID- TEA4>3.0.CO;2-I Goldman, S.R. (2009). Explorations of relationships among learn- ers, tasks, and learning. Learning and Instruction, 19, 451-454. doi:10.1016/j.learninstruc.2009.02.006 Goode, N. & Beckmann, J.F. (2010). You need to know: There is a causal relationship between structural knowledge and con- trol performance in complex problem solving tasks. Intelligence, 38(3), 345-352. doi:10.1016/j.intell.2010.01.001 Goode, N. (2011). Determinants of the control of dynamic sys- tems: The role of structural knowledge (Doctoral thesis). Syd- 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 12 http://dx.doi.org/10.1037/a0021017 http://dx.doi.org/10.4324/9781315806938 http://dx.doi.org/10.1007/s11251-013-9280-7 http://dx.doi.org/10.4324/9781315806723 http://dx.doi.org/10.1016/j.learninstruc.2009.02.024 http://dx.doi.org/10.1080/14640748408402156 http://dx.doi.org/10.1080/14640748708401804 https://www.psychologie.uni-heidelberg.de/ae/allg_en/forschun/dynamis/dynamis_review_08-2005.pdf https://www.psychologie.uni-heidelberg.de/ae/allg_en/forschun/dynamis/dynamis_review_08-2005.pdf http://dx.doi.org/10.1016/j.intell.2008.03.008 http://dx.doi.org/10.1080/02724980143000262 http://dx.doi.org/10.1023/A:1002993406499 http://dx.doi.org/10.2307/1170753 http://dx.doi.org/10.1126/science.1230579 http://dx.doi.org/10.1017/cbo9780511816819 http://dx.doi.org/10.1027/1614-0001/a000055 http://dx.doi.org/10.1177/104687818001100108 http://dx.doi.org/10.1111/bjop.12046 http://cogprints.org/3004/1/Funke_1992_GJP.pdf http://cogprints.org/3004/1/Funke_1992_GJP.pdf http://dx.doi.org/10.1080/13546780042000046 http://dx.doi.org/10.1002/acp.2350090706 http://dx.doi.org/10.1016/j.intell.2015.07.006 http://dx.doi.org/10.1002/(SICI)1098-2736(199901)36:1<39::AID-TEA4>3.0.CO;2-I http://dx.doi.org/10.1002/(SICI)1098-2736(199901)36:1<39::AID-TEA4>3.0.CO;2-I http://dx.doi.org/10.1016/j.learninstruc.2009.02.006 http://dx.doi.org/10.1016/j.intell.2010.01.001 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems ney, Australia: University of Sydney. Retrieved from: http:// hdl.handle.net/2123/8967 Hulshof, C.D. & de Jong, T. (2006). Using just-in-time informa- tion to support scientific discovery learning in a computer-based simulation. Interactive Learning Environments, 14(1), 79-94. doi:10.1080/10494820600769171 Kirschner, P.A., Sweller, J., & Clark, R.E. (2006). Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41(2), 75- 86. doi:10.1207/s15326985ep4102_1 Kluge, A. (2008). Performance assessments with microworlds and their difficulty. Applied Psychological Measurement, 32(2), 156- 180. doi:10.1177/0146621607300015 Kotovsky, K., Hayes, J.R., & Simon, H.A. (1985). Why are some problems hard? Evidence from Tower of Hanoi. Cognitive Psy- chology, 17, 248-294. doi:10.1016/0010-0285(85)90009-x Kröner, S. (2001). Intelligenzdiagnostik per Computersimulation [Intelligence assessment via computer simulation]. Münster: Waxmann. Kröner, S., Plass, J.L., & Leutner, D. (2005). Intelligence as- sessment with computer simulations. Intelligence, 33, 347-368. doi:10.1016/j.intell.2005.03.002 Kretzschmar, A., & Süß, H. M. (2015). A study on the training of complex problem solving competence. Journal of Dynamic Decision Making, 1, 4. doi:10.11588/jddm.2015.1.15455 Lazonder, A.W., Wilhelm, P., & Hagemans, M.G. (2008). The influence of domain knowledge on strategy use during simulation- based inquiry learning. Learning and Instruction, 18, 580-592. doi:10.1016/j.learninstruc.2007.12.001 Lazonder, A.W., Wilhelm, P., & van Lieburg, E. (2009). Un- ravelling the influence of domain knowledge during simulation- based inquiry learning. Instructional Science, 37, 437-451. doi:10.1007/s11251-008-9055-8 Lazonder, A. W. & Harmsen, R. (2016). Meta-Analysis of inquiry- based learning: Effects of guidance. Review of Educational Re- search, 86, 681-718. doi:10.3102/0034654315627366 Leutner, D. (1993). Guided discovery learning with computer- based simulation games: effects of adaptive and non-adaptive instructional support. Learning and Instruction, 3, 113-132. doi:10.1016/0959-4752(93)90011-n Mayer, R.E. (2004). Should there be a three-strikes rule against pure discovery learning?. American Psychologist, 59, 14-19. doi:10.1037/0003-066x.59.1.14 Müller, H. (1993). Komplexes Problemlösen: Reliabilität und Wis- sen [Complex problem solving: Reliability and knowledge]. Bonn: Holos. Osman, M. (2008). Observation can be as effective as action in problem solving. Cognitive Science: A Multi-disciplinary Journal, 32(1), 162-183. doi:10.1080/03640210701703683 Osman, M. (2010). Controlling uncertainty: a review of human be- havior in complex dynamic environments. Psychological Bulletin, 136(1), 65-86. doi:10.1037/a0017815 Preußler, W. (1996). Zur Rolle expliziten und impliziten Wis- sens bei der Steuerung dynamischer Systeme [On the role of explicit and implicit knowledge in controlling dynamic systems]. Zeitschrift für Experimentelle Psychologie, 43, 399-434. Preußler, W. (1998). Strukturwissen als Voraussetzung für die Steuerung komplexer dynamischer Systeme [Structural knowl- edge as a precondition of controlling complex dynamic systems]. Zeitschrift für Experimentelle Psychologie, 45, 218-240. Putz-Osterloh, W. (1981). Über die Beziehung zwischen Testintel- ligenz und Problemlöseerfolg. [The relation between test intelli- gence and problem solving success]. Zeitschrift für Psychologie mit Zeitschrift für angewandte Psychologie, 189(1), 79-100. Putz-Osterloh, W. (1993). Strategies for knowledge acquisition and transfer of knowledge in dynamic tasks. In G. Strube & K. Wender (Eds.) The cognitive psychology of knowledge (pp. 331- 350). Amsterdam, Netherlands: North-Holland/Elsevier Science Publishers. doi:10.1016/s0166-4115(08)x6106-0 Putz-Osterloh, W. & Lüer, G. (1981). Über die Vorhersagbarkeit complexer Problemlöseleistungen durch Ergebnisse in einem In- telligenztest [On the predictability of complex problem solving performance by intelligence test scores]. Zeitschrift für Experi- mentelle und Angewandte Psychologie, 28, 309-334. Raudenbush, S. W. & Bryk, A. S. (2002). Hierarchical linear mod- els: Applications and data analysis method. Newbury Park, CA: Sage. Raudenbush, S. W., Bryk, A. S., Cheong, Y., & Congdon, R. T. (2000). HLM 6: Hierarchical linear and nonlinear modeling. Chicago, IL: Scientific Software International. Schoppek, W. (2002). Examples, rules and strategies in the con- trol of dynamic systems. Cognitive Science Quarterly, 2, 63- 92. Retrieved from: http://www.psychologie.uni-bayreuth.de/ de/documents/ExRulStratScho.pdf Schoppek, W. (2004). Teaching structural knowledge in the control of dynamic systems: Direction of causality makes a difference. In K. D. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th Annual Conference of the Cognitive Science Society. (pp. 1219-1224). Mahwah, NJ: Erlbaum. Snodgrass, J. G. & Corwin, J. (1988). Pragmatics of measur- ing recognition memory: applications to dementia and amne- sia. Journal of Experimental Psychology: General, 117(1), 34. doi:10.1037/0096-3445.117.1.34 Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295- 312. doi:10.1016/0959-4752(94)90003-5 Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Research Review, 22, 123-138. doi:10.1007/s10648-010-9128-5 Raven, J., Raven, J.C., & Court, J. H. (1998). Manual for Raven’s Progressive Matrices and Vocabulary Scales. Section 4: The Advanced Progressive Matrices. San Antonio, TX: Harcourt As- sessment. Snow, R.E. (1986). Individual differences and the design of education programs. American Psychologist, 41, 1029-1039. doi:10.1037/0003-066x.41.10.1029 Snow, R.E. (1989). Towards assessment of cognitive and cona- tive structures in learning. Educational Researcher, 18(9), 8-14. doi:10.2307/1176713 Snow, R.E. & Lohman, D.F. (1989). Implication of cognitive psy- chology for education measurement. In R.L. Linn (Ed.), Educa- tional Measurement (3rd ed., pp. 263-331). New York: Macmil- lan. Snow, R.E. & Yallow, E. (1982). Education and intelligence. In R. J. Sternberg (Ed.), Handbook of Human Intelligence (pp. 493- 586). London, UK: Cambridge University Press. Süß, H.-M. (1996). Intelligenz, Wissen und Problemlösen. Kog- nitive Voraussetzungen für erfolgreiches Handeln bei comput- ersimulierten Problemen [Intelligence, knowledge, and problem solving: Cognitive prerequisites of successful performance in computer-simulated problems]. Lehr- und Forschungstexte Psy- chologie. Göttingen, Germany: Hogrefe. Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2004). Multimedia instructions and cognitive load theory: effects of 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 13 http://hdl.handle.net/2123/8967 http://hdl.handle.net/2123/8967 http://dx.doi.org/10.1080/10494820600769171 http://dx.doi.org/10.1207/s15326985ep4102_1 http://dx.doi.org/10.1177/0146621607300015 http://dx.doi.org/10.1016/0010-0285(85)90009-x http://dx.doi.org/10.1016/j.intell.2005.03.002 http://dx.doi.org/10.11588/jddm.2015.1.15455 http://dx.doi.org/10.1016/j.learninstruc.2007.12.001 http://dx.doi.org/10.1007/s11251-008-9055-8 http://dx.doi.org/10.3102/0034654315627366 http://dx.doi.org/10.1016/0959-4752(93)90011-n http://dx.doi.org/10.1037/0003-066x.59.1.14 http://dx.doi.org/10.1080/03640210701703683 http://dx.doi.org/10.1037/a0017815 http://dx.doi.org/10.1016/s0166-4115(08)x6106-0 http://www.psychologie.uni-bayreuth.de/de/documents/ExRulStratScho.pdf http://www.psychologie.uni-bayreuth.de/de/documents/ExRulStratScho.pdf http://dx.doi.org/10.1037/0096-3445.117.1.34 http://dx.doi.org/10.1016/0959-4752(94)90003-5 http://dx.doi.org/10.1007/s10648-010-9128-5 http://dx.doi.org/10.1037/0003-066x.41.10.1029 http://dx.doi.org/10.2307/1176713 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346 Goode & Beckmann: On the role of guidance in complex, dynamic systems modality and cueing. British Journal of Educational Psychology, 74, 71 - 81. doi:10.1348/000709904322848824 Vollmeyer, R., Burns, B.D., & Holyoak, K. (1996). The impact of goal specificity on strategy use and the acquisition of problem structure. Cognitive Science, 20, 75-100. doi:10.1016/s0364- 0213(99)80003-2 Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex prob- lem solving. More than reasoning? Intelligence, 40, 147-168. doi:10.1080/13546783.2013.844729 10.11588/jddm.2016.1.33346 JDDM | 2016 | Volume 2 | Article 5 | 14 http://dx.doi.org/10.1348/000709904322848824 http://dx.doi.org/10.1016/s0364-0213(99)80003-2 http://dx.doi.org/10.1016/s0364-0213(99)80003-2 http://dx.doi.org/10.1080/13546783.2013.844729 https://journals.ub.uni-heidelberg.de/index.php/jddm/10.11588/jddm.2016.1.33346