Theoretical Contribution The multiple faces of complex problems: A model of problem solving competency and its implications for training and assessment. Andreas Fischer1 and Jonas C. Neubert2 1Department of Psychology, Heidelberg University and 2Institute for Economics, University of Cottbus In this paper, we present a competency model for com- plex problem solving (CPS) by building on the cate- gories of Knowledge, Skills, Abilities, and Other compo- nents (KSAO). We highlight domain-general and domain- specific components in each of these categories, review established conceptualizations of CPS, and present a new model of CPS competency that is meant to provide a starting point for systematic research on training and as- sessment. The model highlights the idea that complex problems differ with regard to the KSAO components they demand from a problem solver and that performance in one problem does not necessarily predict performance in a different problem. Implications for research on the train- ing and assessment of CPS competency are discussed, and a selection of well-established tests for various com- ponents of the KSAO model is provided. Keywords: complex problem solving, problem solving competency, KSAO model The notion that the world around us is developing to-ward complexity has become something of a truism in recent times (see as far back as Weaver, 1948). Modern appliances release us from repetitive daily chores, such as washing the dishes or vacuuming our homes, as well as from routine work tasks, such as checking the usual suspects in a malfunctioning car engine or checking our spelling in a letter. In return, we are facing ever-increasing numbers of situations in which we need to handle these very appli- ances or generally deal with non-routine situations at work. Consequently, an increasing amount of planning and prob- lem solving is required: The dish washer needs to be pro- grammed even when we have lost the manual, and an au- tomotive technician has to analyze and understand an in- creasingly complex array of settings and errors in the elec- tronic control unit of a car (e.g., related to the fine-tuning of an engine). As a result, problem solving competency1 is soaring in importance across occupations (e.g., Neubert, Mainert, Kretzschmar, & Greiff, 2014; Spitz-Oener, 2006). In 2012, one of the most important large-scale assess- ments in education, the OECD’s Programme for Inter- national Student Assessment (PISA), therefore featured creative problem solving alongside its traditional tests of mathematics, science, and literacy. With the help of this additional problem-solving assessment of about 85,000 stu- dents from 44 countries around the world, the OECD tried to establish an empirical basis for suitable policies that could be applied to prepare students for the challenges fac- ing them in the working world (OECD, 2014). Similarly, information networks, such as the Occupational Informa- tion Network (O*NET) of the United States Department of Labor (http://www.onetonline.org/), introduced skills such as critical thinking or complex problem solving into their repertoire to account for the changing requirements in today’s jobs, thereby including new requirements in their standardized overviews of critical knowledge, skills, and abilities (National Center for O*NET Development, 2009). While the world has been taking a closer look at how humans interact with complex problems, many conceptual questions have emerged: What are complex problems and, relatedly, what does it take to solve them? What are the major differences between someone who can solve an arbi- trary problem effectively and someone who cannot? How can people who lack problem solving skills be trained? How can such skills be transferred to problems from different do- mains? Questions such as these are related to the idea of Complex Problem Solving competency (CPS competency) and have important implications for training and assess- ment. In the paper at hand, we review different ways of conceptualizing CPS competency, and we present a new model that can be applied to clarify the unique contri- butions of Knowledge, Skills, Abilities, and Other compo- nents (KSAO) to the solution of different kinds of complex problem situations. Thereby, we elaborate on a suggestion made by Funke, Fischer & Holt (2015) to view problem solving competency “as a bundle of skills, knowledge and abilities, which are required to deal effectively with com- plex non-routine situations in different domains.” The descriptor KSAO is borrowed from the industrial and organizational psychology literature where it is typ- ically used to describe the requirements of different work situations with the help of competencies, a task that seems similar to our quest for a closer link between CPS com- petency and actual complex problem situations (see, e.g., Campion et al., 2011, for more details on KSAO models in industrial and organizational psychology or Peterson et al., 2001, for an exemplary and comprehensive KSAO model). In order to systematically examine the nature of a CPS competency from a coherent and unifying perspective, we have to specify what different complex problems have in common and, thus, what it takes to solve them. On the ba- sis of Fischer, Greiff and Funke’s (2012) theoretical frame- work for the process of solving complex problems, we de- Corresponding author: Andreas Fischer, Heidelberg University, Hauptstr. 47-51 69117 Heidelberg (Germany). e-mail: andreas.fischer@psychologie.uni- heidelberg.de 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 1 mailto:andreas.fischer@psychologie.uni-heidelberg.de mailto:andreas.fischer@psychologie.uni-heidelberg.de http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems rived a model of CPS competency that connects the com- plex problems of real life to the (psychological) constructs contributing to effective problem solving within and across a wide range of domains. In doing so, we offer a foundation for future research and for translating such research into practice. What are complex problems and why are they so hard to handle? According to Duncker’s (1945) seminal definition, a prob- lem arises when a person has a goal but does not know how to achieve it. This lack of knowledge might refer to the representation of the problem or it might refer to its goal-oriented application (Dörner, 1979). In other words, there might be no known operator2 for reaching one’s goals, or an operator might be known to exist, but the operator might not be applicable in the current situation.3 In the example of a car’s electronic control unit, I might not have a good idea of how to deal with a problem concerning the timing of the ignition (i.e., lacking a suitable operator), or I might know that certain tools should be applied, but I might not have the necessary tools available. In both cases, a problem arises because I do not have a viable path by which to reach the desired goal state of repairing the ignition system. A problem is complex when multiple highly interrelated elements have to be considered in order to derive a solu- tion (Dörner, 1996; Fischer et al., 2012; Weaver, 1948). In a similar vein, the process of solving a problem can be considered complex when it involves multiple highly interrelated elements (e.g., multiple search spaces; Klahr & Dunbar, 1988; Fischer et al., 2012). Characteristically, as emphasized by Funke (2003), in the process of solving complex problems, there are multiple interrelated goals to consider (a feature called polytely), for example, finding a solution to a car’s ignition problem that balances both the cost of additional parts and the time needed to solve the problem with their help. Typically, solving complex prob- lems also involves dealing with overwhelming amounts of information that is more or less relevant for solving the problem (i.e., complexity), for example, readouts from the engine control unit related to ignition timing and fuel in- jection but also service intervals, driving behavior, and the financial resources of the customer. Furthermore, multiple effects of actions need to be considered (i.e., interconnect- edness) because changes in the engine control unit can re- sult in emission problems, legal risks, or problems with a supervisor to give just a few examples. Finally, complex problems are typically characterized by incomplete knowl- edge about the status quo or the effect of interventions (i.e., intransparency) and the need for as well as the pos- sibility of dynamically adapting one’s course of action at future points in time (i.e., dynamic decision making; cf. Fischer, Holt & Funke, 2015), for example, when trying to find optimal ignition parameters in a car racing event when one needs to decide whether to repair the current engine or switch to a new one. The KSAO-model of CPS competency In the current paper, we argue that the features of com- plex problems give rise to a set of characteristic demands (Funke, 2001). More specifically, we propose the KSAO- model of problem solving competency, consisting of dif- ferent categories (Knowledge, Skills, Abilities, and Other components) and emphasizing both domain-general and domain-specific components in each category (see Funke et al., 2015), for a more general perspective on CPS com- petency). Figure 1 illustrates how the components of CPS com- petency are assumed to be related to CPS performance. In our view domain-generality and domain-specificity are endpoints of a continuum with content-neutral cognitive structures (e.g., working memory) on one end and very spe- cific knowledge stored in long-term memory on the other (Funke et al., 2015). Please note that the effect of the KSAO components on CPS performance will depend on various features of both the problems and the problem solvers (e.g., Fischer et al., 2012; Funke, 1991; Hundert- mark, Holt, Fischer, Said, & Fischer, 2015). For instance, the relation between intelligence and CPS is known to de- pend on the situation’s transparency (cf. Putz-Osterloh & Lüer, 1981) and on the problem solver’s prior knowledge (Leutner, 2002). In the paper at hand, however, we will elaborate primarily on the ellipses in Figure 1. That is, we will let future publications provide a detailed answer to the question of moderator variables (the interested reader should refer to Funke, 2003, or Süß, 1996, for a preliminary summary of empirical findings). 1) Knowledge One of the most obvious and central determinants of CPS performance is declarative knowledge (e.g., about means and ends; cf. Funke, 2003). There is a long history of experiments and investigations into the role of knowledge and its relations to various aspects of (complex) problem solving (e.g., Funke, 1985, Kersting & Süß, 1995, Mor- ris & Rouse, 1985, see Funke, 1992, and Süß, 1996, for overviews). For example, Kersting and Süß (1995) inves- tigated the construction of a content-valid knowledge test for a specific computer-simulated problem, differentiating knowledge from the simple recognition of relations to the prediction of strength and direction of numerical relations. Relatedly, there is an even longer tradition in the area of industrial and organizational psychology and the analysis of work tasks and requirements also dedicated to the clas- sification and comparison of knowledge in different (work) situations. For example, the classification of knowledge in the O*NET content model is structured along domains of knowledge, such as Business and Management (e.g., areas of knowledge related to clerical procedures or sales and marketing), Manufacturing and Production (e.g., knowl- edge related to food production), or health services (e.g., medical knowledge, see Constanza, Fleishman, & Marshall- Mies, 1999). This differentiation of knowledge require- ments from the area of industrial and organizational psy- 1 In the remainder of this paper we will use the term CPS com- petency to emphasize that - in contrast to the simple research paradigms applied in laboratory research on problem solving (see Funke, 2003) - most problems in real life are complex (to varying degrees). 2 Operators are actions that can be applied to transform a sit- uation into a different one. They can be separated from tac- tics (i.e., chains of operators) and strategies (i.e., more general guidelines for when to apply which tactic or operator; cf. Güss, Tuason, & Orduña, 2015). Similarly, heuristics guide the prob- lem solver toward specific operators in a given problem situation (e.g., Gigerenzer & Gaissmaier, 2011). 3 A lack of knowledge or specificity concerning goals (Dörner, 1979) can also be subsumed under this kind of means-end- analysis-problem if knowledge acquisition is considered a means towards the end of solving the problem (cf. Fischer, 2015a). 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 2 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Figure 1. The KSAO model of CPS competency: Solid lines represent direct causal influences, dotted lines represent moderating effects. Ellipses represent the different categories of CPS competency, and solid rectangles represent manifest phenomena. We expect different components within each category to be relevant for different problem solvers or in problem situations (because of various kinds of moderator variables). With the exception of Knowledge, the moderating influence of KSAO categories – and their higher order interactions – have been omitted for visual clarity. chology can be helpful for understanding, differentiating, and comparing complex problem situations in very dif- ferent work environments. For example, a comparison of knowledge requirements in different complex problems re- lated to the engine control unit of a car can build on a prob- lem’s relation to different knowledge areas (e.g., problems requiring knowledge about ignition parameters or emission regulations). Knowing about the different kinds of knowl- edge involved in solving different problems is important for characterizing jobs, for predicting the performance of problem solvers (e.g., job applicants), and for training pur- poses. A small fraction of knowledge can be useful in a very broad range of complex problem situations: For in- stance, knowing about the definition of a complex problem and its characteristic features (i.e., CPS-related concepts; Schoppek & Fischer, 2015), or knowledge about a range of exploration strategies can foster a systematic approach to solving such a problem. That is, knowing that com- plex problems typically involve one or more dimensions of evaluating potential goal states may help to explicate dif- ferences in goal states and to detect and resolve conflicts between the various dimensions of goals. Human wisdom – knowledge and deep understanding of fundamental goals and how to reach them (Fischer, 2015b), that is, knowl- edge about the “fundamental pragmatics of life” (Baltes & Smith, 1990) – can also be considered helpful for solving many kinds of complex problems because of its conceptual relation to balancing multiple interests, contextualizing ac- tion, and managing uncertainty (Fischer, 2015a). However, most of the knowledge that is relevant for solv- ing complex problems is highly domain-specific, for exam- ple, as knowledge about the analysis of engine problems is most certainly irrelevant for many cases of career plan- ning (see also Tricot & Sweller, 2013). More specifically, basic knowledge about relevant problem characteristics is required in most (complex) problem settings, for example, knowledge about functional sub-elements, such as those that control fuel injection and spark timing is required for the application of basic exploration strategies, such as VOTAT (see below). In turn, this basic knowledge is strongly linked to the domain at hand (e.g., the different sub-components of a car). Furthermore, outside of very restricted and formalized problem-solving settings, prior knowledge of available operators is helpful or might even be a prerequisite for (complex) problem-solving attempts; when exploring problems in cars without electronic engine control units, “connecting the service computer” is not an option for obtaining information about an ignition prob- lem, but it might be a prerequisite in other cases (e.g., when dealing with problems related to a modern car’s emis- sions). Again, knowledge of available operators is strongly linked to the problem’s domain: If a person has no idea about the availability of an operator, he or she will not be able to use it. Research comparing experts and novices offers impor- tant insights in this regard (e.g., Chi et al., 1981, 1982). For example, domain-specific expertise seems to alter the ways in which (complex) problems are conceptualized on a very basic level (e.g., leading to the use of different cate- gories of problems; see also below). What seems clear is the paramount role of knowledge for solving complex problems (see Greeno, 1997, for a critical view on the general notion of separating knowledge from the situation it is utilized in). Consequently, differentiating the knowledge requirements of different complex problems appears to be a worthwhile endeavor for CPS research. 2) Skills Besides knowledge about concepts, solving complex prob- lems requires skills that will allow the knowledge to be put into practice (cf. Anderson, 1987). In contrast to “knowing 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 3 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems that”, it is also important to “know how” to apply heuris- tics, strategies, tactics, or operations the right way and at the right moment in time (Dörner, 1986; 1996; Güss et al., 2015; see Süß, 1996, for a discussion). In KSAO models from the area of industrial and or- ganizational psychology, skills are closely associated with the level of proficiency needed to perform a given task (e.g., Peterson et al., 2001). For example, the O*NET content model mentioned above features a skill taxonomy that differentiates between two “higher-order constructs”, namely, basic skills (facilitating learning, e.g., reading, writing, and critical thinking) and cross-functional skills (facilitating performance across domains, e.g., social skills and resource management skills, see Mumford, Peterson, & Childs, 1999, and Peterson et al., 2001, for more details). With respect to CPS competency, these skill taxonomies offer a way to compare different complex problem situa- tions with regard to their requirements in terms of skills. Complex problems in the context of repairing car engines differ in terms of their reliance on written materials (i.e., requiring a certain amount of reading comprehension) or in the degree to which theybenefit from a high level of social skills (e.g., exploring customer needs). Some skills can be applied or adapted to a broad range of domains, such as social skills (Weis & Conzelmann, 2015), instance-based decision making (Gonzalez, Lerch, & Lebiere, 2003; Dutt & Gonzalez, 2015) or scientific proce- dures such as the systematic testing of hypotheses. These kinds of skills can be applied in the case of the automotive mechanic but also when looking for factors that influence career development. Other skills are related to a smaller set of domains, for example, calculating the cost of changing a part in the car on the basis of the cost of labor, the cost of the part, and other general costs such as rent or even a single domain (e.g., operating a specialized tool to read out the engine parameters from the car). The link or dis- sociation between declarative knowledge (see above) and procedural skills has been the subject of empirical research (see Süß, 1996, for an overview). What seems important here is the consideration of both declarative knowledge and the translation into problem solving via skills when trying to analyze complex problem performance in given situa- tions as they are potentially dissociated (see also Nicko- laus, 2011). That is, assessing declarative knowledge in a given domain of CPS can lead to a false impression of CPS competency when procedural aspects are not accounted for (as in the experiments conducted by Berry & Broadbent, 1988, see also Süß, 1996). Again, the link to research on expertise offers valuable insights here. For example, research on differences in prob- lem solving between experts and novices has shown that an understanding of important characteristics of a prob- lem situation is one of the key factors that differentiates expert problem solvers from beginners (i.e., experts focus on the problems’ “deep structure” as opposed to superfi- cial features, e.g., Larkin, McDermott, Simon, & Simon, 1980, Chi, Feltovich & Glaser, 1981, Chi, Glaser & Rees, 1982). Furthermore, research on the potentially adverse ef- fect of educational interventions in the case of misaligned prior experience in learners (the so-called expertise rever- sal effect; e.g., Kalyuga, Ayres, Chandler, & Sweller, 2003; Rey & Fischer, 2013) warns conceptions of accounting for prior knowledge and skills that are too simplistic. Natu- rally, such (procedural) knowledge is strongly bound to a domain of application, even though we expect that exper- tise in one field of complex problems will be helpful for shaping a specific (and more or less helpful) perspective in a different domain (see Beckmann & Guthke, 1995, for a more nuanced view). Skills that are relevant in current assessments of CPS – such as the (more or less systematic) application of the strategy “vary potentially influencing factors in isolation” to find out about relations between problem elements (e.g., the VOTAT strategy; Wüstenberg, Stadler, Hautamäki, & Greiff, 2014) – fall somewhere in between the two extremes, being relevant in a range of problem situations, for exam- ple, in the context of scientific experiments from a vari- ety of domains (e.g., Scherer & Tiemann, 2012). At the same time, there are certainly limits to the applicability of this type of skill to other cases of complex problems (see also Tricot & Sweller, 2013). The automotive technician is well-advised not to apply that kind of strategy to explore the dynamics of frustration in her marriage because the reversibility of actions is rather low (even though the situ- ation exhibits features of a complex problem, e.g., intrans- parency, multiple goals, interrelatedness, and dynamics). Furthermore, the recognition of suitable problem situations for the application of the operator is already strongly de- pendent on domain-specific knowledge structures (i.e., pro- cedural knowledge related to the structure of the problem, i.e., the previously mentioned role of expertise in identify- ing the relevant cues in a situation). Taken together, different complex problem-solving situ- ations will require the problem solver to apply very differ- ent skills. And similar to declarative knowledge, there is a broad body of research that has explored skills across a wide range of settings from entrepreneurship (e.g., Gustafs- son, 2006) to car mechanics and office management (Nick- olaus, 2011). Nonetheless, we see skills as an important element when looking at CPS and its assessment. Relat- edly, the question of the skills that are relevant across com- plex problems, their influence, transferability, and domain- specific aspects seem promising roads for future research (e.g., classes of skills that connect complex problems across different domains). 3) Abilities The utilization of knowledge and skills to deal with a com- plex problem situation generally rests on the fundamental abilities of the problem solver. Abilities can be under- stood as “enduring capacities for performing a wide range of different tasks” (Peterson et al., 2001, p.457). Many abilities can be assumed to be relevant for successfully handling complex problem situations, and in the following, we highlight the role of some exemplary ones. Compara- bly, the O*NET content model features workers’ abilities as part of their framework, building strongly on the work of Fleishman and the related taxonomy and measurement system (the Fleishman-Job Analysis Survey; Fleishman, 1992; Fleishman, Constanza, & Marshall-Mies, 1999). In this classification, abilities are classified along four higher- order dimensions: cognitive abilities (e.g., deductive rea- soning), psychomotor abilities (e.g., wrist-finger speed), physical abilities (e.g., stamina), and sensory abilities (e.g., peripheral vision; see Peterson et al., 2001, and Fleishman et al., 1999, for details). With regard to CPS, there are certain abilities that, al- though they have been the focus of a wide range of empiri- cal research, still deserve special attention from researchers and practitioners. On a basal level, general intelligence and the latent factors that are commonly used to define it 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 4 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems are important prerequisites for solving problems efficiently (Süß, 1999). The importance of intelligence for (complex) problem solving comes as no surprise as definitions of intel- ligence have often included problem solving as a prominent part of the definition (Sternberg, 1982). For instance, complex problems by definition require a large amount of information to be considered simultane- ously (Fischer, 2015a) and thus require the capacity to store and process information (i.e., memory). In the ex- ample of the ignition system, being able to keep in mind the characteristics of the engine control unit (e.g., fuel in- jection settings, ignition timing) is a prerequisite for han- dling the problem situation systematically. If the problem solver lacks the memory capacity to store these basic solu- tion characteristics, a recourse to more heuristic strategies might be necessary, thereby completely changing the inter- action with a complex problem situation. In the same vein, higher levels of reasoning ability can be required to transfer knowledge and strategies to a new problem situation or to analyze the suitability of a specific operator for a specific complex problem (e.g., Süß, 1996; Wittmann & Hattrup, 2004). More specifically, only problem solvers with a cer- tain level of reasoning ability will be able to apply the nec- essary basic (cognitive) operations to a given problem. Süß (1996) provided a comprehensive collection of experimental findings concerning how the role of intelligence depends on various problem-solving characteristics, including, for ex- ample, specific predictions that were based on the novelty and the difficulty of the problem (i.e.,the increased impor- tance of intelligence in the medium range; Raaheim, 1974, Hussy, 1985, see Süß, 1996). Also by definition, problems require deviations from business-as-usual. Thus, creativ- ity with regard to option generation or divergent thinking is also conceptually related to CPS (Kretzschmar, Neu- bert, Wüstenberg, & Greiff, 2016). If the route to the desired goal state is already crystal clear, we are not faced with a real problem situation but rather a routine activity (Funke, 2003). Luckily for CPS research, a host of research has investigated the roles of working memory capacity, rea- soning ability, and other basic abilities in educational set- tings (e.g., Adey, Csapó, Demetriou, Hautamäki, & Shayer, 2007). Consequently, their roles in acquiring new knowl- edge and expertise in specific fields and even some examples of complex problems are well researched (e.g., Ackerman, 1992). Similarly, the role of intelligence for CPS compe- tency has been the target of empirical research throughout the history of CPS research (e.g., Kretzschmar et al., 2016). Besides these general abilities that can be considered necessary or helpful across all complex problem situations, there are also abilities that are relevant for complex prob- lems of a certain kind only: For example, some complex problems are characterized by a large degree of time pres- sure, and thus, cognitive speed is an issue for such prob- lems, whereas other problems require the ability to work adeptly with one’s hands (i.e., the availability of operators depends on a specific degree of psychomotor ability). As in the case of skills (being more or less relevant across complex problem situations), we also see the potential of looking at ways to organize complex problems according to the abilities required for solving them (e.g., domains of complex problems that share similar requirements in terms of reasoning ability or physical strength). Furthermore, the consideration of basic abilities is also relevant when exploring the malleability of complex problem solving per- formance. If solving a specific type of complex problems depends to a large extent on a specific ability (e.g., spa- tial reasoning), the effectiveness of training interventions directed toward advancing problem-solving will be under- estimated if interindividual differences in basic abilities are not taken into account (see e.g., the studies by Ackerman, 1992, on air traffic controllers). That is, if a certain kind of training strengthens the knowledge and skills needed to handle a specific complex problem but participants fail to handle it because their ability levels are insufficient (e.g., a lack of hand-eye coordination), the training might be prematurely dismissed. 4) Other As the world of complex problems is as heterogeneous as our world at large, some other factors deserve considera- tion in relation to CPS. The “Other”-category of the KSAO model is meant to be a category for all the requirements that can potentially arise in CPS situations but are not contained in the previously mentioned categories (e.g., hav- ing a license to practice as a doctor or being able to handle emotional stress). The explicit inclusion of an “Other”- category thereby serves as a route for integration and as a reminder of the factors that are potentially at least as important as the ones presented above. Most important, having a problem implies the frustra- tion of goal achievement, and thus, frustration tolerance is, by definition, an important factor in every instance of CPS (e.g., Funke et al., 2015). In a similar vein, a posi- tive attitude toward problems is an important prerequisite for solving them (D’Zurilla & Nezu, 2007) – for instance, given that a problem can be solved, viewing the problem as inevitable, challenging, and solvable is clearly prefer- able to viewing it as something undesirable, threatening, or that should not have happened in the first place. As solv- ing complex problems requires a systematic and unbiased consideration of options as well as a monitoring of conse- quences, cognitive reflection (Toplak, West, & Stanovich, 2011) – a construct at the border between cognitive ability and cognitive style – might also foster CPS in general (e.g., Donovan, Güss, & Naslund, 2015). It is related to avoid- ing cognitive biases beyond intelligence and a wide range of other variables (Toplak et al., 2011). Further, as initial assumptions about complex problems are always incom- plete and often false, openness to experience and learning motivation – in addition to a sufficient amount of achieve- ment motivation – might also be helpful to varying degrees (cf. Greiff & Neubert, 2014). Besides these highly domain- general aspects, domain-specific motivation, interest, and willingness are important aspects that are involved in every attempt to solve a problem. Furthermore, there is a host of influencing factors relevant in some complex problem situ- ations that lie outside the scope of cognitive considerations (e.g., formal qualifications). For example, the availability of operators might depend on formal qualifications or fi- nancial resources, thereby completely altering the problem situations for a subgroup of problem solvers. In summary, the KSAO model of CPS competency offers a way to systematically look at complex problem situations and highlights relevant categories of prerequisites for prob- lem solving (see Figure 1). At the same time, the model also points toward the need to include a broad array of components within each of these categories to capture an accurate picture of CPS competency. Even more, due to its alignment with models from the area of industrial and organizational psychology, the KSAO model also points to- ward possible ways to link CPS research to the insights that 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 5 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems have been gained in specific fields of application. How is the KSAO model related to previous conceptualizations of CPS competency? In the research literature, one of the most prominent po- sitions is to assume that performance in complex problem situations primarilly depends on a small set of domain- general skills that determine problem-solving performance across different problems (e.g., Greiff, Wüstenberg, & Funke, 2012). If this assertion of a stable set of CPS skills is accurate, a reliable and valid assessment of this set of skills could be used to estimate a person’s level of competency in dealing with complex problem situations in general (e.g., handling complex problems related to career development, leadership, and the domain of mechatronics engineers; Neubert, Mainert, Kretzschmar, & Greiff, 2015). An alternative perspective questions this conception of CPS competency and points toward the multitude of influ- ences and requirements in educational and everyday prob- lem situations (e.g., Tricot & Sweller, 2013). This per- spective implies that the competency to handle complex problems successfully depends to a large degree on domain- specific knowledge and expertise that differs across com- plex problems from different domains. That is, proponents of this second perspective highlight the heterogeneity of requirements that complex problem solvers face and the specificity of knowledge. A third and somewhat conciliatory perspective – and the foundation of our KSAO model – regards CPS as a product of a combination of domain-general facets that are relevant across complex problems (e.g., intelligence) and domain- specific facets (e.g., problem-specific knowledge; e.g., Süß, 1996, 1999) with the importance of these elements differing in accordance with the problem situation at hand (Fischer, 2015a; Funke et al., 2015). The third perspective thereby points toward an array of influences and constructs that work together when individuals deal with complex situ- ations, but this perspective simultaneously acknowledges the relevance of constructs involved across complex prob- lems (see also Funke, 2010; Süß, 1999). The KSAO model includes certain highly domain- general constructs (e.g., intelligence), but it also empha- sizes that the relevance of these constructs differs between problems. Furthermore, it acknowledges the importance of domain-specific knowledge, skills, and even some abilities for solving specific complex problems. Table 1 presents an exemplary list of problems that might occur while work- ing on the fuel ignition system of a car and highlights the role of KSAO components for analyzing and solving these problems. While the strategy of pressing some random buttons in the case of the dish washer might actually lead to some improvement in our understanding and, subsequently, the quality of the dish washing, we surely hope our automo- tive technicians have some more elaborate problem-solving strategies at their disposal, as a trial-and-error strategy for knowledge acquisition seems rather unsuitable. The com- plexity of the car engine problem might otherwise lead to very unfortunate situations where problematic side effects of operators are recognized at high speed on a motorway long after the car has been returned to the customer. It is easy to see how the picture becomes fairly complicated if situational and personal variables are taken into account. Implications No matter which position is taken on how to map CPS competency to performance or to the specific requirements of complex problem situations, CPS competency implies everything a person needs to solve the complex problem(s) at hand (e.g., cognitive abilities, skills, self-regulation, mo- tivation, knowledge of appropriate strategies, and more; Weinert, 1999; Greiff, Wüstenberg et al., 2013; Schoppek & Fischer, 2015). However, the different perspectives have different implications for CPS competency and for how it is related to CPS performance and to the various factors this performance depends on. For instance, according to the KSAO model, the mere concept of CPS competency itself does not imply that these factors (the KSAO components) are empirically correlated (Funke et al., 2015; Schoppek & Fischer, 2015). More specifically, different KSAO components are assumed to make unique contributions to performance in several com- plex problem situations. In line with this implication, Wittmann and Süß (1999) reported that problem-specific knowledge and reasoning ability made significant unique contributions to performance in three heterogeneous com- plex problems. Also, Greiff & Neubert (2014) reported that reasoning, problem solving skills (measured by the MicroDYN test, see Funke, 2010; Fischer, 2015a), and com- puter anxiety each uniquely contributed to the prediction of Grade Point Average. Not all of the components of the KSAO model are likely to be correlated (in a way that would allow for a single underlying CPS factor to emerge), but of course, different components of the KSAO model can be assessed reliably as there are well-established, reli- able, and valid tests for several components of the KSAO model (see Table 2 for an exemplary list of tests). With regard to the requirements of problems, we expect a representative sampling of problem situations to reveal fundamental differences between groups of problems with regard to the KSAO components required – a heteroge- neous set of homogenous clusters (cf. Fischer, 2015a). In line with this implication, empirical studies have repeat- edly shown that the correlations between measures of per- formance in multiple heterogeneous complex problems are rather weak and not substantially different from zero (e.g., Schaub, 1990; Süß, 1999), a finding that might imply that they tap into different subsets of CPS-related constructs (Schoppek & Fischer, 2015). Please note that performance in multiple homogenous problems tends to converge on a latent construct that is separable from other potential con- founding variables (e.g., reasoning ability) and is related to external criteria (e.g., school grades). For example, Süß (1996) reported that performances in multiple trials of managing a simulated tailorshop were substantially cor- related. Similar results have been reported for exploring and regulating multiple complex systems based on formal frameworks (e.g., Fischer et al., 2015; Greiff, Fischer et al., 2013; Kretzschmar, et al., 2016). Another implication of the KSAO model is that the ef- fects of various KSAO components on CPS performance are not constant across different complex problems, but rather, such effects are variable because of moderating vari- ables (e.g., features of the problem, the problem solver, or the problem environment; cf. Funke, 1991). For instance, the correlation between intelligence and performance in the complex problem of managing a tailorshop is known to de- pend on the transparency of the system structure (Putz- Osterloh & Lüer, 1981). Some implications of the KSAO model have been less 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 6 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Table 1. Exemplary application of the KSAO model of CPS competency to the fuel ignition problem. Problem Facet Domain-generality and transfer Knowledge Which settings of the electronic engine control unit are related to the emissions of the engine? Finding the right settings requires declarative (world) knowl- edge of typical cues, such as emission parameters, e.g., from having prior experience with similar devices. low (world knowledge relevant in similar situations) Alternatively, declarative knowledge about a suitable strategy might be a helpful point of departure for exploration. high (exploration strategy can be applied in a range of problems) Skills Finding out which combination of settings results in an accept- able combination of accelera- tion, gasoline consumption, and emission parameters. Different combinations of input settings need to be evaluated. To this end, specific skills come in handy, such as checking the results of different parameter combinations(e.g., procedural knowledge related to theevaluation of charts and tables). low to medium (specific skills re- lated to the evaluation of re- sults) When trying out different combinations, the task also requires the application of a range of motor skills (e.g., hand-eye coor- dination, for example, whenmanually adjusting the settings of the engine in an old car without an electronic engine control unit). high (motor coordination, hand- eye coordination are relevant in most problems involving manual manipulations) Abilities Which combination exhibited the best overall quality? The evaluation of results requires the recall of information from long term memory as well as keeping in mind the outcomes of different attempts (i.e., requiring a certain amount of working memory). high (working memory capacity is relevant in all problems that require the comparison of out- comes) Other How should the problem solver react when the testing equip- ment shows only erratic behavior (e.g., a malfunctioning connec- tor between computer and car)? If an attempt at problem solving goes wrong, and the testing equipment exhibits erratic behavior, a positive attitude toward the problem and the belief that a solution is possible is neces- sary in order to decide not to give up. high (positive attitude toward problem solving should help in addressing difficult problems in general) researched but could be put to the test in future studies. One of these implications is that the correlation between performance in two complex problem situations depends on their similarity regarding the required KSAO components and moderating variables: In general, performance should be correlated if both problems pose similar requirements (and are presented to similar samples of problem solvers in similar environments). More specifically, it should be pos- sible to predict the correlation of performance in two com- plex problems by the degree of overlap between the KSAO components that are required. If a person with certain abilities, skills, knowledge, and other features solves two reliable problems, systematic differences in performance should reflect differences in the requirements of the prob- lems. Analogously, if two people solve the same reliable problem and differ with regard to a single KSAO com- ponent only (e.g., intelligence), systematic differences in performance should reveal that this KSAO component is required to solve the problem. How can those who do it right be identified? After introducing ther KSAO model, it seems vital to dis- cuss some implications for the assessment of CPS compe- tency in more detail. Naturally, conceptualizing CPS com- petency as described above also has implications for the identification of interindividual differences in CPS. For many (highly domain-specific or comparatively domain-general) components of the KSAO model, well- established tests have been proposed, that can serve as a point of departure for the assessment of the different KSAO components. Table 2 lists some of the most impor- tant constructs that we have mentioned so far (along with suggested tests for each aspect). It is important to note that the components highlighted in the different categories of CPS competency in Table 2 are by no means exhaustive. Accounting for the heterogeneity of different complex problems and the challenges they pose to problem solvers with different levels of proficiency will require to look at both the problem- and domain-specific aspects of complex problems, as well as efforts to connect different complex problems and problem solvers. With regard to assessments of CPS competency as a whole, some researchers have proposed that multiple heterogeneous problems should be applied (Süß, 1999; Wittmann & Süß, 1999, Neubert et al., 2014), whereas others have proposed that multiple homogenous problems should be applied (Greiff et al., 2012). It is important to note that in both cases, the idea is to measure the level of competency as the average level of performance across multiple problems – and to view problem solving compe- tency as a “reflective construct” that is reliably indicated by performance in different problems. This approach as- sumes that performance in different complex problems is correlated (because of the common influence of CPS com- petency), an idea that seems to be at odds with some of the empirical evidence reported above (e.g., a lack of cor- relations between heterogeneous problems). However, a lack of correlation between heterogeneous problems in empirical studies does not tell us anything about the number of competent persons who solved all kinds of problems, whether a subgroup of the problems 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 7 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Table 2. The following table summarizes some examples of Knowledge, Skills, Abilities, and Other constructs relevant for CPS. KSAO component Type Suggested test Description World Knowledge (about prob- lem solving & fundamental pragmatics of life) K Berlin Wisdom Paradigm There are a range of assessment instruments target- ing (declarative) knowledge, both, on a general level (i.e., general world knowledge), such as the Bochum Knowledge Test (BOWIT, Hossiep & Schulte, 2008) Domain-specific knowledge (occupational) K Knowledge tests developed in the area of vocational education, Task specific work analyses or the Berlin Wisdom Paradigm (Baltes & Staudinger, 2000) and also domain-specific knowledge tests rang- ing from classical knowledge assessments in primary education to office administration and engine prob- lems (see for example Nickolaus, 2011, for an overview Domain-specific knowledge (ed- ucational) K Classical tests from (large-scale) educational testing in the area of vocational education). Adapt plans and hypotheses to feedback S MicroDYN / MicroFIN Compared to the assessment of (declarative) knowl- edge, the case is more difficult for skills. Devel- opments towards the computer-based assessment of competencies in vocational education might offer Domain-specific problem solv- ing skills (vocational education) S Domain-specific skill assessment (see Nickolaus, 2011) some relief in terms of providing assessment instru- ments directed at domain-specific skills for a range of vocations (Nickolaus, 2011, see Neubert et al., 2015). Similarly, the assessment of skills in the Domain-specific problem solv- ing skills (school assessments) S Domain-specific skill assessment (see OECD, 2013) classical domains of school (e.g., mathematics and science education) have seen a rise in importance (e.g., in the context of large-scale assessments, such as PISA, OECD, 2014). Reasoning A BIS Assessing basic human abilities is one of the bedrocks of modern psychological assessment, so instruments Creativity A BIS, Unusual uses test, Option generation indicating for example an individuals’ general mental ability or working memory capacity are readily Implicit Learning A Artificial Grammar available and well-established (e.g., the Berlin Struc- ture Intelligence test, Jäger, 1984). Cognitive Reflecton O CRT Like the complex problems in our world, the assess- ment of the “Other”-category is very heterogeneous. Openness to experience O NEO-FFI While there are established assessment instruments for some of the potentially relevant factors falling Learning motivation O FAM under this category, such as those connected to per- sonality dimensions (e.g., openness to experience is Achievement motivation O AMI included in the five-factor model of personality), other constructs lack appropriate and well-validated Frustration-tolerance O DTS assessment instruments (e.g., ethical reasoning; how- ever see Lind, 2000). Note. KSAO component: exemplary component for the different categories of influencing factors within the KSAO model of CPS competency; Type: related category of the respective component in the KSAO model: K = Knowledge, S = Skills, A = Abilities, O = Other influencing factors; Suggested test: MicroDYN / MicroFIN: MicroDYN / MicroFIN test of CPS skills (Neubert et al., 2014); BIS = Berlin Intelligence Structure test (Jäger, Süß, & Beauducel, 1997); Unusual uses test (Guilford, Merrifield, & Wilson, 1958); Option generation (Johnson & Raab, 2003); Artificial Grammar(Mackintosh, 1998)); CRT (Toplak et al., 2011); NEO-FFI = Assessment of the five factor model of personality (Costa & McCrae, 1992), FAM (Rhein- Rheinberg, Vollmeyer, & Burns, 2001), AMI = Achievement Motivation Inventory (Schuler, Thornton, Frintrup & Müller-Hanson, 2004), DTS = Disstress Tolerance Scale (Simons & Gaher, 2005). 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 8 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems had similar requirements for some of the problem solvers, or “how” participants succeeded or failed at solving them (i.e., which were the critical components; see Schoppek & Fischer, 2015). A lack of correlations implies only that knowing that a person solved one problem is not informa- tive for predicting whether that person will solve the other problems as well. In our view, a competent person is able to solve a wide range of problems by virtue of the components of the KSAO model (whether or not performance in these prob- lems is correlated in samples of CPS novices). Different problems will pose different demands on problem solvers, but a person with high levels on each of the KSAO compo- nents is likely to solve a wide range of problems (e.g., not only those from the domain of fuel ignition but also those related to career development). Each component of CPS competency might be mea- sured reliably, but CPS competency itself – as suggested by Schoppek and Fischer (2015) as well as by Funke et al. (2015) – does not seem to be a reflective construct (i.e., a latent construct that produces high levels on each of the components). Based on the empirical evidence reported above, we argue that CPS competency might be better conceptualized as a set of KSAO components or – if a sin- gle score is preferred – as a formative construct (defined by high levels on each of the components; cf. Edwards, 2010, Bollen & Bauldry, 2011). In future studies, depending on the goals of researchers or practitioners, KSAO components can be studied in iso- lation, or they can be aggregated into a formative con- struct. Furthermore, building on a coherent framework of CPS competency facilitates a combination of insights from different studies via the KSAO model even when the com- plex problems are very different (e.g., requiring different amounts of prior knowledge in a domain while sharing a reliance on social skills and creativity). Discussion A host of problems in our daily lives are complex. We identified a wide range of Knowledge, Skills, Abilities, and Other factors, that are relevant for solving a wide range of complex problems (see Table 2), but we also highlighted domain-specific aspects that are important for certain sub- sets of complex problems only. Further, we emphasized that the domain-generality of each component might de- pend on several moderating variables (see Figure 1). Previous discussions of CPS competency have either pro- posed a unitary conception of CPS, highlighting the role of a single set of important skills for handling all complex problems (perspective 1) or denied the educational rele- vance of domain-general factors (perspective 2)4 . In this paper, we argued for a middle ground between these two extremes (perspective 3) and proposed a model of CPS competency that might offer a point of departure for ex- plaining differences and commonalities in performance for different complex problems. Beyond a focus on the reliable assessment of important components, there are also more practical arguments for the use of competency models in CPS research, such as the promotion of communication in applied settings, the facil- itation of a developmental perspective, or the potential to strategically align individual resources with organizational needs (see Campion et al., 2011). The heterogeneity of requirements included in the KSAO model of CPS competency (see Table 2) as well as the em- pirical findings reported above should have made clear that a unitary conception of CPS is probably not a realistic op- tion, even just for capturing the differences between prob- lem solvers. Nevertheless, (cor-)relations between the com- ponents of CPS competency should be explored in future research in order to determine which components are best suited for assessment purposes in different settings (and which components are redundant in situations of interest). At the same time, the new model of competency of- fers a way to compare and identify the requirements that overlap between the different domains of complex problems as these can be compared beyond domain-specific concep- tions of competency (e.g., with regard to how to handle multiple goals or dynamically developing situations; e.g., Schoppek & Fischer, 2015). Relatedly, for most practical purposes, it will not be feasible or necessary to assess all of the components of the KSAO model. Instead, it might be sufficient to assess the subset of KSAO-component that is most relevant to the problems of interest. The KSAO model thereby allows a systematic search of relevant com- ponents to be conducted, for example, when handling com- plex engine problems. In contrast to previous attempts to connect the domain- general and domain-specific elements of CPS (e.g., Süß, 1996), the KSAO model allows for the integration and specification of a broader range of additional factors (e.g., including factors such as self-regulation in the context of specific domains of problems or manual skills related to a group of work tasks), thereby going beyond the combi- nation of intelligence and knowledge, for example, as pro- posed by Süß (1996). These additional factors might be important for better predicting (or systematically foster- ing) human performance across a wide range of complex problems. Last but not least, while previous research on CPS has primarily tried to identify the overlap or distinctiveness of CPS skills with regard to intelligence or reasoning ability, the KSAO model offers a route by which theoretical and empirical integration can occur: Skills and abilities are dif- ferent categories, and both of them are relevant for solving complex problems – in most (but not all) circumstances. Intelligence and reasoning – among others – are important prerequisites for many instances of CPS and are thus part and parcel of CPS competency. Nonetheless, there will be other components of CPS competency that are much more relevant than intelligence for the handling of subgroups of complex problems (e.g., strategic knowledge; Strohschneider & Güß, 1999). Limitations and concerns The proposed KSAO model of CPS competency offers some guidance in terms of factors to account for when looking at complex problem solving. It also can be applied as a framework by which to integrate different factors that are relevant in sets of complex problem situations. Nonethe- less, there are limitations to both the model itself and the explanations and examples we offered here. The roots of the KSAO model in an applied setting of assessment, namely, that of industrial and organizational psychology, at the one hand, support a closer connection 4 The argumentation is a bit more nuanced with respect to the perspective 2 camp, as the relevance of basic abilities (e.g., work- ing memory capacity, intelligence) for complex problem solving is well acknowledged. 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 9 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems and transfer of insights between CPS research and the re- spective fields of application. For example, working with a similar distinction of important factors will facilitate the transferability of insights (e.g., from the work and task analysis literature) into the field of CPS research. More specifically, we expect benefits from the use of established methods for characterizing complex problem situations and their requirements in specific work environments. At the same time, this close connection to an applied setting also weakens the link between CPS competency and classical cognitive architectures of human functioning. That is, the model offers no explanation for or specificytion of the processes that integrate the different constituting factors, their relation to each other, or a coherent underly- ing model of human cognition and action (e.g., specifying the link between basic abilities and knowledge). For in- stance, we did not specify in detail the set of moderator variables that are likely to determine which complex prob- lems are correlated with each other (as a result of similar requirements regarding the KSAO components) or with the components of the KSAO model (but see Funke, 2003). A topic that is up for debate is whether this integration with cognitive science can be achieved in future research within the framework of the KSAO model or whether different models that are more closely aligned with classical cog- nitive architectures5 or more detailed theories of human action in the work place (e.g., the “Handlungsregulations- theorie” of Winfried Hacker, 1973) are necessary to account for these drawbacks. Furthermore, the nature of the KSAO model of CPS competency builds on the (culturally embedded) concept of “competency”. In the context of vocational education, Brockmann, Clarke, and Winch (2008) differentiate be- tween a “skills-based model” of competency in the Anglo- Saxon context and a “knowledge-based model” in Ger- many, the Netherlands, and France. While the “skills- based model” focuses on learning outcomes and their cer- tification in view of a specific work task without a close connection to the knowledge base, the “knowledge-based model” is oriented toward understanding the inputs to learning in order to build a broader conceptualization of (vocational) education (i.e., “Berufsbildung”, with an em- phasis on a holistic view of education). We can only spec- ulate on whether and how this distinction will be relevant for the discussion of CPS competency, although we sus- pect that a narrow and unitary view on CPS competency makes sense only in the context of a “skills-based model” of competencies. Highlighting the role of domain-specific components (e.g., knowledge about dish washers and fuel ignition systems) and broader components (e.g., human wisdom; see above) will offer a way to differentiate between the two perspectives in CPS research and will facilitate an exploration of consequences for learning and instruction. What seems important to us is the necessity to account for various assumptions that underlie the construction and application of assessment instruments by researchers from various backgrounds, as well as the mental construction of situations by individual problem solvers (e.g., Roth, 1998). In our view, the KSAO model of CPS competency offers a point of departure for future discussions along these lines. Finally, there is the question of the practical relevance of the proposed KSAO model: The notion of a modular CPS competency that integrates the requirements of spe- cific problem situations naturally leads to the question of the empirical relevance of its sub-components (and in this way, the utility of a modular conception of CPS compe- tency itself). That is, the question arises as to whether the integration of additional constructs (in addition to the ele- ments already included in unitary conceptions of CPS or in- telligence theories) will offer additional value in predicting CPS performance at large – and under what circumstances (i.e., the external validity of tests might well differ between different samples; e.g., Fischer et al., 2015). The literature on the empirical relevancy of the sub-components of intel- ligence in predicting success across a range of job-related performance indicators warns of expectations that are too enthusiastic to say the least (see e.g., Schmidt, Ones, & Hunter, 1992). Time and again, the power of specific sub- components has been shown to exhibit only marginal in- crements over indicators of general mental ability when predicting job or training performance (e.g., Brown, Le, & Schmidt, 2006). Similar findings might be expected for the case of CPS competency. For the purposes of training CPS competency (see Kret- zschmar & Süß, 2015), the notion of a modular CPS compe- tency provides a way to look at the actual reasons for why some individuals fail when attempting to handle specific groups of complex problems, where they are potentially in need of further training or support, and in which area to look for further expertise when trying to set up such inter- ventions (e.g., teaching explicit and domain-specific knowl- edge vs. selecting only individuals above a certain thresh- old of memory capacity vs. training the transfer of skills to a new domain; see McClelland, 1973, for similar earlier ar- guments in favor of competency models). Naturally, there is the need to actually deliver empirical evidence for this additional utility of a modular CPS competency, and we can only point to future research for that purpose. Conclusion According to Dörner (1996), who initiated research on CPS in the late 1970s, there are manifold failures that can be observed in attempts to handle complex problems. Corre- sponding to the numerous sources of failures, we proposed a multifaceted model of CPS competency, consisting of Knowledge, Skills, Abilities, and Other influencing compo- nents. We highlighted domain-general and domain-specific components in each category in order to demonstrate the large numbers of requirements that problem solvers face in complex problem situations and we discussed implications for research, assessment and training. As Dörner (1986) noted, acknowledging complexity is an important prereq- uisite for the proper assessment and training of CPS com- petency. This is not going to be a walk in the park, but given the high relevance of CPS for modern life, it may well be worth the effort. Acknowledgements: This research was self-funded. Declaration of conflicting interests: The authors de- clare that the research was conducted in the absence of any commercial or financial relationships that could be constructed as a potential conflict of interest. 5 It is interesting to note that one of the origins of competency models in industrial and organizational psychology is McClel- land’s (1973) work, which is directly related to the assessment of basic abilities such as intelligence. 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 10 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Author contributions: The authors contributed equally to this paper. Handling editor: Jan Rummel Copyright: This work is licensed under a Creative Com- mons Attribution-NonCommercial-NoDerivatives 4.0 In- ternational License. Citation: Fischer, A. & Neubert, J.C. (2015). The multiple faces of complex problems: A model of prob- lem solving competency and its implications for training and assessment. Journal of Dynamic Decision Making, 1, 6. doi: 10.11588/jddm.2015.1.23945 Received: 29 September 2015 Accepted: 16 February 2016 Published: 25 February 2016 References Ackerman, P. L. (1992). Predicting individual differences in com- plex skill acquisition: dynamics of ability determinants. Jour- nal of Applied Psychology, 77(5), 598–614. doi: 10.1037/0021- 9010.77.5.598 Adey, P., Csapó, B., Demetriou, A., Hautamäki, J., & Shayer, M. (2007). Can we be intelligent about intel- ligence? Educational Research Review, 2(2), 75–97. doi: 10.1016/j.edurev.2007.05.001 Anderson, J. R. (1987). Skill acquisition: Compilation of weak-method problem situations. Psychological Review, 94(2), 192. doi: 10.1037/0033-295X.94.2.192 Baltes, P. B., & Smith, J. (1990). Weisheit und Weisheitsentwick- lung: Prolegomena zu einer psychologischen Weisheitstheorie. [Wisdom and how it develops: Framework of a psychological theory of wisdom]. Zeitschrift für Entwicklungspsychologie und pädagogische Psychologie, 22(2), 95-135. Baltes, P. B., & Staudinger, U. M. (2000). Wisdom: a metaheuris- tic (pragmatic) to orchestrate mind and virtue toward excellence. American Psychologist, 55(1), 122-136. doi: 10.1037/0003- 066X.55.1.122 Beckmann, J. F., & Guthke, J. (1995). Complex problem solving, intelligence, and learning ability. In P. A. Frensch & J. Funke (Eds.), Complex Problem Solving: The European Perspective (pp. 177–200). Hillsdale, NJ: Erlbaum. Berry, D. C., & Broadbent, D. E. (1988). Interactive tasks and the implicit-explicit distinction. British Journal of Psychology, 79(2), 251–272. doi: 10.1111/j.2044-8295.1988.tb02286.x Bollen, K. A., & Bauldry, S. (2011). Three Cs in measurement models: Causal indicators, composite indicators, and covariates. Psychological Methods, 16(3), 265-284. doi: 10.1037/a0024448 Brockmann, M., Clarke, L., & Winch, C. (2008). Knowledge, skills, competence: European divergences in vocational edu- cation and training (VET) - the English, German and Dutch cases. Oxford Review of Education, 34(5), 547–567. doi: 10.1080/03054980701782098 Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training per- formance? International Journal of Selection and Assessment, 14(2), 87–100. doi: 10.1111/j.1468-2389.2006.00336.x Campion, M. A., Fink, A. A., Ruggeberg, B. J., Carr, L., Phillips, G. M., & Odman, R. B. (2011). Doing competencies well: Best practices in competency modeling. Personnel Psychology, 64(1), 225–262. doi: 10.1111/j.1744-6570.2010.01207.x Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Cat- egorization and representation of physics problems by ex- perts and novices. Cognitive Science, 5(2), 121–152. doi: 10.1207/s15516709cog0502_2 Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (Vol. 1, pp. 7-75). Hillsdale, NJ: Erlbaum. Constanza, D. G., Fleishman, E. A., & Marshall-Mies, J. C. (1999). Knowledge. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret, & E. A. Fleishman (Eds.), An occupational infor- mation system for the 21st century: The development of O*NET (pp. 71–90). Washington, D.C.: APA. doi: 10.1037/10313-005 Costa, P. T., & MacCrae, R. R. (1992). Revised NEO personality inventory (NEO PI-R) and NEO five-factor inventory (NEO FFI): Professional manual. Odessa, FL: Psychological Assessment Re- sources. D’Zurilla, T. J., & Nezu, A. M. (2007). Problem-solving ther- apy: A positive approach to clinical intervention. New York, NY: Springer. Donovan, S.J., Güss, C.D., & Naslund, D. (2015): Improving dy- namic decision making through training and self-reflection. Judg- ment and Decision Making, 10(4), 284-295. Dörner, D. (1979). Problemlösen als Informationsverarbeitung [Problem solving as information processing]. Stuttgart: Kohlhammer. Dörner, D. (1986). Diagnostik der operativen Intelligenz [Assess- ment of operative intelligence]. Diagnostica, 32(4), 290–308. Dörner, D. (1996). The logic of failure: Recognizing and avoiding error in complex situations. New York, NY: Perseus. Duncker, K. (1945). On problem-solving. Psychological Mono- graphs, 58(5), 1–113. doi: 10.1037/h0093599 Dutt, V. & Gonzalez, C. (2015). Accounting for outcome and pro- cess measures in dynamic decision-making tasks through model calibration. Journal of Dynamic Decision Making, 1 ,2. doi: 10.11588/jddm.2015.1.17663 Edwards, J. R. (2010). The fallacy of formative measure- ment. Organizational Research Methods.14(2), 370–388. doi: 10.1177/1094428110378369 Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4(1), 19-42. doi: 10.7771/1932-6246.1118 Fischer, A., Greiff, S., Wüstenberg, S., Fleischer, J., Buchwald, F., & Funke, J. (2015). Assessing analytic and interactive aspects of problem solving competency. Learning and Individual Differences, 39, 172-179. doi: 10.1016/j.lindif.2015.02.008 Fischer, A., Holt, D. V., & Funke, J. (2015). Promoting the grow- ing field of dynamic decision making. Journal of Dynamic Deci- sion Making, 1, 1. doi: 10.11588/jddm.2015.1.23807 Fischer, A. (2015a). Assessment of problem solving skills by means of multiple complex systems – validity of finite automata and lin- ear dynamic systems (Doctoral dissertation). urn:nbn:de:bsz:16- heidok-196898 Fischer, A. (2015b). Wisdom - The answer to all the questions re- ally worth asking. International Journal of Humanities and Social Science, 5(9), 73-83. urn:nbn:de:bsz:16-heidok-197863 Fleishman, E. A. (1992). Fleishman-Job Analysis Survey (F-JAS). Potomac, MD: Management Research Institute. Fleishman, E. A., Constanza, D. G., & Marshall-Mies, J. C. (1999). 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 11 http://dx.doi.org/10.1037/0021-9010.77.5.598 http://dx.doi.org/10.1037/0021-9010.77.5.598 http://doi.org/10.1016/j.edurev.2007.05.001 http://doi.org/10.1016/j.edurev.2007.05.001 http://dx.doi.org/10.1037/0033-295X.94.2.192 http://dx.doi.org/10.1037/0003-066X.55.1.122 http://dx.doi.org/10.1037/0003-066X.55.1.122 http://doi.org/10.1111/j.2044-8295.1988.tb02286.x http://doi.org/10.1037/a0024448 http://doi.org/10.1080/03054980701782098 http://doi.org/10.1080/03054980701782098 http://doi.org/10.1111/j.1468-2389.2006.00336.x http://doi.org/10.1111/j.1744-6570.2010.01207.x http://dx.doi.org/10.1207/s15516709cog0502_2 http://dx.doi.org/10.1207/s15516709cog0502_2 http://dx.doi.org/10.1037/10313-005 http://doi.org/10.1037/h0093599 http://doi.org/10.1/jddm.2015.1.17663 http://doi.org/10.1/jddm.2015.1.17663 http://doi.org/10.1177/1094428110378369 http://doi.org/10.1177/1094428110378369 http://dx.doi.org/10.7771/1932-6246.1118 http://dx.doi.org/10.7771/1932-6246.1118 http://doi.org/10.1016/j.lindif.2015.02.008 http://doi.org/10.11588/jddm.2015.1.23807 http://nbn-resolving.de/urn:nbn:de:bsz:16-heidok-196898 http://nbn-resolving.de/urn:nbn:de:bsz:16-heidok-196898 http://nbn-resolving.de/urn:nbn:de:bsz:16-heidok-197863 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Abilities. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret, & E. A. Fleishman (Eds.), An occupational infor- mation system for the 21st century: The development of O*NET (pp. 175–195). Washington, DC: APA. doi: 10.1037/10313-010 Funke, J. (1985). Problemlösen in komplexen computersimulierten Realitätsbereichen [Problem solving in complex computer- simulated domains of reality]. Sprache & Kognition, 4, 113-129. Funke, J. (1991). Solving complex problems: Exploration and con- trol of complex systems. In R. J. Sternberg & P. A. Frensch (Eds.), Complex problem solving: Principles and mechanisms (pp. 185-222). Hillsdale, NJ: Erlbaum. Funke, J. (1992). Dealing with dynamic systems: Research strat- egy, diagnostic approach and experimental results. German Jour- nal of Psychology, 16, 24-43. Funke, J. (2001). Dynamic systems as tools for analysing hu- man judgement. Thinking & Reasoning, 7(1), 69–89. doi: 10.1080/13546780042000046 Funke, J. (2003). Problemlösendes Denken [Problem-solving Thinking]. Stuttgart: Kohlhammer. Funke, J. (2010). Complex problem solving: a case for com- plex cognition? Cognitive Processing, 11(2), 133–142. doi: 10.1007/s10339-009-0345-0 Funke, J., Fischer, A., & Holt, D. (2015). Competencies for com- plexity: Problem solving in the 21st century. In E. Care, P.Griffin, & M. Wilson (Eds.), Assessment and Teaching of 21st Century Skills (Volume 3). Manuscript submitted for publication. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62(1), 451–482. doi: 10.1146/annurev-psych-120709-145346 Gonzalez, C., Lerch, J. F., & Lebiere, C. (2003). Instance-based learning in dynamic decision making. Cognitive Science, 27(4), 591–635. doi: 10.1016/S0364-0213(03)00031-4 Greiff, S., Fischer, A., Wüstenberg, S., Sonnleitner, P., Brunner, M., & Martin, R. (2013). A multitrait–multimethod study of as- sessment instruments for complex problem solving. Intelligence, 41(5), 579–596. doi: 10.1016/j.intell.2013.07.012 Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new assessment perspective. Applied Psychological Measurement, 36(3), 189–213. doi: 10.1177/0146621612439620 Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapó, B. (2013). Complex problem solving in educational con- texts—Something beyond g: Concept, assessment, measurement invariance, and construct Validity. Journal of Educational Psy- chology, 105(2), 364–379. doi: 10.1037/a0031856 Greiff, S., & Neubert, J. C. (2014). On the relation of com- plex problem solving, personality, fluid intelligence, and academic achievement. Learning and Individual Differences, 36, 37–48. doi: 10.1016/j.lindif.2014.08.003 Greeno, J. G. (1997). On claims that answer the wrong questions. Educational Researcher, 26(1), 5-17. doi: 10.3102/0013189X026001005 Guilford, J. P., Merrifield, P. R., & Wilson, R. C. (1958). Unusual uses test. Orange, CA: Sheridan Psychological Services. Güss, C. D., Tuason, M. T., & Orduña, L.V. (2015). Strate- gies, tactics, and errors in dynamic decision making in an Asian sample. Journal of Dynamic Decision Making, 1, 3. doi: 10.11588/jddm.2015.1.13131 Gustafsson, V. (2006). Entrepreneurial Decision-Making: Individ- uals, tasks and cognitions. Cheltenham, UK: Elgar. Hacker, W. (1973). Allgemeine Arbeits- und Ingenieurpsychologie: psychische Struktur und Regulation von Arbeitstätigkeiten [Gen- eral industrial and engineering psychology–Mental structure and regulation of working activities]. Berlin: VEB Deutscher Verlag der Wissenschaften. Hossiep, R., & Schulte, M. (2008). BOWIT. Bochumer Wis- senstest [BOWIT. Bochum Knowledge Test]. Göttingen: Hogrefe. Hundertmark, J., Holt, D. V., Fischer, A., Said, N., & Fischer, H. (2015). System structure and cognitive ability as predictors of performance in dynamic system control tasks. Journal of Dy- namic Decision Making, 1, 5. doi: 10.11588/jddm.2015.1.26416 Hussy, W. (1985). Komplexes Problemlösen - Eine Sackgasse? [Complex problem solving - A dead end?]. Zeitschrift für Exper- imentelle und Angewandte Psychologie, 32, 55–74. Jäger, A. O. (1984). Intelligenzstrukturforschung: Konkurrierende Modelle, neue Entwicklungen, Perspektiven. [Structural research on intelligence: Competing models, new developments, perspec- tives]. Psychologische Rundschau, 35(1), 21–35. Jäger, A. O., Süß, H. M., & Beauducel, A. (1997). Berliner Intelligenzstruktur-Test (BIS-Test): Form 4 [The Berlin Intel- ligence Structure Test: Form 4]. Göttingen: Hogrefe. Johnson, J. G., & Raab, M. (2003). Take the first: Option- generation and resulting choices. Organizational Behav- ior and Human Decision Processes, 91(2), 215-229. doi: 10.1016/S0749-5978(03)00027-X Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. doi: 10.1207/S15326985EP3801_4 Kersting, M., & Süß, H. M. (1995). Kontentvalide Wissensdiag- nostik und Problemlösen: Zur Entwicklung, testtheoretischen Be- gründung und empirischen Bewährung eines problemspezifischen Diagnoseverfahrens. [Content valid diagnostics of knowledge and problem solving: Development, testtheoretical establishment and empirical proof of a problem specific diagnostical instrument.] Zeitschrift für Pädagogische Psychologie, 9(2), 83-93. Klahr, D., & Dunbar, K. (1988). Dual space search dur- ing scientific reasoning. Cognitive Science, 12(1), 1-48. doi: 10.1207/s15516709cog1201_1 Kretzschmar, A., Neubert, J. C., Wüstenberg, S., & Greiff, S. (2016). Construct validity of complex problem solving: A com- prehensive view on different facets of intelligence and school grades. Intelligence, 54, 55-69. doi: 10.1016/j.intell.2015.11.004 Kretzschmar, A. & Süß, H.-M. (2015). A study on the training of complex problem solving competence. Journal of Dynamic Decision Making, 1, 4. doi:10.11588/jddm.2015.1.15455 Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208(4450), 1335–1342. doi: 10.1126/sci- ence.208.4450.1335 Leutner, D. (2002). The fuzzy relationship of intelligence and prob- lem solving in computer simulations. Computers in Human Be- havior, 18(6), 685-697. doi: 10.1016/S0747-5632(02)00024-9 Lind, G. (2000). Review and appraisal of the moral judgment test (MJT). Psychology of morality and democracy and education, 1-15. Mackintosh, N. J. (1998). IQ and human intelligence. Oxford, UK: Oxford University Press. McClelland, D. C. (1973). Testing for competence rather than for "intelligence". American Psychologist, 28(1), 1–14. doi: 10.1037/h0034092 Morris, N. M., & Rouse, W. B. (1985). Review and evaluation of empirical research in troubleshooting. Human Factors: Journal of the Human Factors and Ergonomics Society, 27(5), 503-530. doi: 10.1177/001872088502700502 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 12 http://dx.doi.org/10.1037/10313-010 http://doi.org/10.1080/13546780042000046 http://doi.org/10.1080/13546780042000046 http://doi.org/10.1007/s10339-009-0345-0 http://doi.org/10.1007/s10339-009-0345-0 http://doi.org/10.1146/annurev-psych-120709-145346 http://doi.org/10.1146/annurev-psych-120709-145346 http://doi.org/10.1016/S0364-0213(03)00031-4 http://doi.org/10.1016/j.intell.2013.07.012 http://doi.org/10.1177/0146621612439620 http://doi.org/10.1037/a0031856 http://doi.org/10.1016/j.lindif.2014.08.003 http://doi.org/10.1016/10.3102/0013189X026001005 http://doi.org/10.1016/10.3102/0013189X026001005 http://doi.org/10.11588/jddm.2015.1.13131 http://doi.org/10.11588/jddm.2015.1.13131 http://doi.org/10.11588/jddm.2015.1.26416 http://dx.doi.org/10.1016/S0749-5978(03)00027-X http://dx.doi.org/10.1016/S0749-5978(03)00027-X http://doi.org/10.1207/S15326985EP3801_4 http://doi.org/10.1207/s15516709cog1201_1 http://doi.org/10.1207/s15516709cog1201_1 http://dx.doi.org/10.1016/j.intell.2015.11.004 http://doi.org/10.11588/jddm.2015.1.15455 http://doi.org/10.1126/science.208.4450.1335 http://doi.org/10.1126/science.208.4450.1335 http://doi.org/10.1016/S0747-5632(02)00024-9 http://doi.org/10.1037/h0034092 http://doi.org/10.1037/h0034092 http://doi.org/10.1177/001872088502700502 http://doi.org/10.1177/001872088502700502 http://dx.doi.org/10.11588/jddm.2015.1.23945 Fischer & Neubert: The multiple faces of complex problems Mumford, M. D., Peterson, N. G., & Childs, R. A. (1999). Basic and cross-functional skills. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret, & E. A. Fleishman (Eds.), An occupational information system for the 21st century: The de- velopment of O*NET (pp. 49–69). Washington, DC: APA. doi: 10.1037/10313-004 National Center for O*NET Development. (2009). New and emerging occupations of the 21st century: Updating the O*NET- SOC Taxonomy. Neubert, J. C., Kretzschmar, A., Wüstenberg, S., & Greiff, S. (2014). Extending the assessment of complex problem solv- ing to finite state automata: Embracing heterogeneity. Eu- ropean Journal of Psychological Assessment, 31, 181-194. doi: 10.1027/1015-5759/a000224 Neubert, J. C., Mainert, J., Kretzschmar, A., & Greiff, S. (2015). The assessment of 21st century skills in industrial and organiza- tional psychology: Complex and collaborative problem solving. Industrial and Organizational Psychology, 8(2), 238–268. doi: 10.1017/iop.2015.14 Nickolaus, R. (2011). Die Erfassung fachlicher Kompetenzen und ihrer Entwicklungen in der beruflichen Bildung – Forschungs- stand und Perspektiven [Assessment of specialist competencies and their development in professional education - state of the art and perspectives]. In O. Zlatkin-Troitschanskaia (Ed.), Stationen Empirischer Bildungsforschung (pp. 331–351). Wiesbaden: VS Verlag für Sozialwissenschaften. doi: 10.1007/978-3-531-94025- 0_24 OECD. (2013). PISA 2012 Assessment and analytical framework: Mathematics, reading, science, problem solving and financial lit- eracy. Paris: OECD Publishing. doi: 10.1787/9789264190511-en OECD. (2014). PISA 2012 results: Creative problem solv- ing (Volume V). Paris; France: OECD Publishing. doi: 10.1787/19963777 Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K. Y., & others. (2001). Understanding work using the Occupational Information Network (O* NET): Implications for practice and research. Personnel Psychology, 54(2), 451–492. doi: 10.1111/j.1744-6570.2001.tb00100.x Putz-Osterloh, W., & Lüer, G. (1981). The predictability of complex problem solving by performance on an intelligence test. Zeitschrift für Experimentelle und Angewandte Psychologie, 28(2), 309-334. Raaheim, K. (1974). Problem solving and intelligence. Oslo, Nor- way: Universitetsforlaget. Rey, G. D., & Fischer, A. (2013). The expertise reversal effect con- cerning instructional explanations. Instructional Science, 41(2), 407-429. doi: 10.1007/s11251-012-9237-2 Rheinberg, F., Vollmeyer, R., & Burns, B. D. (2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leis- tungssituationen 12 (Langversion, 2001) [A questionnaire to as- sess current motivation in learning situations]. Diagnostica, 2, 57-66. doi: 10.1026//0012-1924.47.2.57 Roth, W.-M. (1998). Situated cognition and assessment of com- petence in science. Evaluation and Program Planning, 21(2), 155–169. doi: 10.1016/S0149-7189(98)00004-4 Schaub, H. (1990). Die Situationsspezifität des Problemlösev- erhaltens. [Situational specifity of problem solving behavior]. Zeitschrift für Psychologie mit Zeitschrift für angewandte Psy- chologie, 198(1), 83-96. Scherer, R., & Tiemann, R. (2012). Factors of problem-solving competency in a virtual chemistry environment: The role of metacognitive knowledge about strategies. Computers & Edu- cation, 59(4), 1199–1214. doi: 10.1016/j.compedu.2012.05.020 Schmidt, F. L., Ones, D. S., & Hunter, J. E. (1992). Personnel selection. Annual Review of Psychology, 43(1), 627–670. doi: 10.1146/annurev.ps.43.020192.003211 Schoppek, W. & Fischer, A. (2015). Complex problem solv- ing—single ability or complex phenomenon? Frontiers in Psy- chology, 6, 1669. doi: 10.3389/fpsyg.2015.01669 Schuler, H., Thornton, G.C., Frintrup, A., & Mueller-Hanson, R. (2004). Achievement Motivation Inventory (AMI). Göttingen, Bern, New York: Huber. Simons, J. S., & Gaher, R. M. (2005). The Distress Tolerance Scale: Development and validation of a self-report measure. Mo- tivation and Emotion, 29(2), 83-102. Spitz-Oener, A. (2006). Technical change, job tasks, and ris- ing educational demands: Looking outside the wage struc- ture. Journal of Labor Economics, 24(2), 235–270. doi: 10.1086/jole.2006.24.issue-2 Sternberg, R. J. (1982). Handbook of human intelligence. New York, NY: Cambridge University. Süß, H.-M., Kersting, M., & Oberauer, K. (1991). Intelligenz und Wissen als Prädiktoren für Leistungen bei computersimulierten komplexen Problemen [Intelligence and knowledge as predictors of performance in computer-simulated complex problems]. Diag- nostica, 37(4), 334-352. Süß, H.-M. (1996). Intelligenz, Wissen und Problemlösen. Kog- nitive Voraussetzungen für erfolgreiches Handeln bei computer- simulierten Problemen [Intelligence, knowledge and problem solv- ing: Cognitive prerequisites for successful behavior in computer- simulated problems]. Göttingen: Hogrefe. Süß, H.-M. (1999). Intelligenz und komplexes Problemlösen [Intelli- gence and complex problem solving]. Psychologische Rundschau, 50(4), 220–228. doi: 10.1026//0033-3042.50.4.220 Strohschneider, S., & Güss, D. (1999). The fate of the Moros: A cross-cultural exploration of strategies in complex and dynamic decision making. International Journal of Psychology, 34(4), 235-252. Toplak, M., West, R. F., & Stanovich, K. E. (2011). The Cogni- tive Reflection Test as a predictor of performance on heuristics- and-biases tasks. Memory and Cognition, 39, 1275–1289. doi: 10.3758/s13421-011-0104-1 Tricot, A., & Sweller, J. (2013). Domain-specific knowledge and why teaching generic skills does not work. Educational Psychol- ogy Review, 26(2), 1–19. doi: 10.1007/s10648-013-9243-1 Weaver, W. (1948). Science and complexity. American Scientist, 36, 536-547. doi: 10.1016/B978-0-08-097086-8.25094-0 Weinert, F. E. (1999). Konzepte der Kompetenz. [Concepts of competence]. Paris: OECD. Weis, S. & Conzelmann, K. (2015). Social Intelligence and Com- petencies. In J. D. Wright, International Encyclopedia of the Social & Behavioral Sciences (2nd ed.). (pp. 371–379). Oxford: Elsevier. doi: 10.1016/B978-0-08-097086-8.25094-0 Wittmann, W., & Hattrup, K. (2004). The relationship be- tween performance in dynamic systems and intelligence. Sys- tems Research and Behavioral Science, 21(4), 393–409. doi: 10.1002/sres.653 Wittmann, W., & Süß, H.-M. (1999). Investigating the paths between working memory, intelligence, knowledge, and complex problem-solving performances via Brunswik symmetry. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual differences: Process, trait, and content determinants (pp. 77–108). Washington, D.C.: APA. doi: 10.1037/10315-004 Wüstenberg, S., Stadler, M., Hautamäki, J., & Greiff, S. (2014). The role of strategy knowledge for the application of strategies in complex problem solving tasks. Technology, Knowledge and Learning, 19, 127–146. doi: 10.1007/s10758-014-9222-8 10.11588/jddm.2015.1.23945 JDDM | 2015 | Volume 1 | Article 6 | 13 http://dx.doi.org/10.1037/10313-004 http://dx.doi.org/10.1037/10313-004 http://doi.org/10.1027/1015-5759/a000224 http://doi.org/10.1027/1015-5759/a000224 http://doi.org/10.1017/iop.2015.14 http://doi.org/10.1017/iop.2015.14 http://dx.doi.org/10.1007/978-3-531-94025-0_24 http://dx.doi.org/10.1007/978-3-531-94025-0_24 http://dx.doi.org/10.1787/9789264190511-en http://dx.doi.org/10.1787/19963777 http://dx.doi.org/10.1787/19963777 http://dx.doi.org/10.1111/j.1744-6570.2001.tb00100.x http://dx.doi.org/10.1007/s11251-012-9237-2 http://dx.doi.org/10.1026//0012-1924.47.2.57 http://doi.org/10.1016/S0149-7189(98)00004-4 http://doi.org/10.1016/j.compedu.2012.05.020 http://doi.org/10.1146/annurev.ps.43.020192.003211 http://doi.org/10.1146/annurev.ps.43.020192.003211 http://doi.org/10.3389/fpsyg.2015.01669 http://doi.org/10.1086/jole.2006.24.issue-2 http://doi.org/10.1086/jole.2006.24.issue-2 http://doi.org/10.1026//0033-3042.50.4.220 http://doi.org/10.3758/s13421-011-0104-1 http://doi.org/10.3758/s13421-011-0104-1 http://doi.org/10.1007/s10648-013-9243-1 http://dx.doi.org/10.1016/B978-0-08-097086-8.25094-0 http://dx.doi.org/10.1016/B978-0-08-097086-8.25094-0 http://doi.org/10.1002/sres.653 http://doi.org/10.1002/sres.653 http://dx.doi.org/10.1037/10315-004 http://doi.org/10.1007/s10758-014-9222-8 http://dx.doi.org/10.11588/jddm.2015.1.23945