Quality of Research Based on Design-Based Research Approach 6 _______________________________________________________________________________________________________________________ Global Education Review is a publication of The School of Education at Mercy University, New York. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License, permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. Citation: Schäfers, Maria Sophie, & Wegner, Claas. (2024). Quality of Research Based on Design-Based Research Approach: Using an Example from Early Childhood Talent Research. Global Education Review, 11 (1), 6-22. Quality of Research Based on Design-Based Research Approach: Using an Example from Early Childhood Talent Research Maria Sophie Schäfers Claas Wegner Abstract Over the past decades, the demand for scientific research to link theory and practice so that innovative solutions can be developed through research has steadily increased (Klees & Tillmann, 2015; Sandoval & Bell, 2004). In this context, the design-based research (DBR) approach was developed, which examines the research object and the research setting from multiple perspectives through a cyclical- iterative process (Shavelson et al., 2003; Reinmann, 2005). The connection between research and practice is not only relevant for school contexts, but also for early childhood education (Schäfers & Wegner, 2021a). Therefore, the project “Kleine BegInNa” utilizes the DBR approach. A preliminary examination revealed that early promotion has a great influence on the children's later school performance, yet that there is not any test to survey their competences. However, this is essential in order to be able to offer appropriate support. In the prototyping phase, a scientific talent test for pre-school aged children was developed. This test was validated in several cycles in the assessment phase (Schäfers & Wegner, 2022a). The last validity test for the elicitation of the internal structure by an exploratory factor analysis shows that both the loadings and the screeplot, output one factor. This one factor solution explains 43.56% of the variance. This may be the “scientific talent” factor, which needs to be verified in further studies and by confirmatory factor analysis. While Reinmann (2022) has established comprehensive standards for science and methodology in DBR, we also focus on what extent conventional quality criteria for qualitative and quantitative research can be applied to a DBR approach. Keywords Early childhood education, “Kleine BegInNa,” scientific talent test, design-based research, DBR, quantitative study, exploratory factor analysis, Germany, pre-school age Introduction For a long time, school research and thus research for practice has faced a major barrier (Juuti & Lavonen, 2006): either the researchers focused their studies on problems that were too detached from practical work and were thus not relevant for teachers or pedagogical specialists, or the studies produced strategies for problems that could not be implemented in practice for various reasons (e.g., lack of time, lack of materials, too much effort, assuming ideal-typical learning groups). In both cases, there has been a lack of links between theory and practice, so that they did not support each other (Schmiedebach & Wegner, 2021). This, in turn, meant that the intended synergy effects were thus not present. However, the need for change in the school context is particularly urgent. Not least because of the results of international comparative studies such as PISA (Programme for International Student Assessment) or TIMSS (Third International Mathematics and Science Study), as they have Quality of Research Based on Design-Based Research Approach 7 given cause for concern and fostered the desire to reform the school landscape (Gundlach, 2003). Nevertheless, innovation is not the first term to be associated with school-related development (Asbrand, 2009). Oftentimes, the only innovations implemented in schools are ideas from politics and economy, rather than from educational research (Dehmel, 2018; Reinmann, 2005). Thus, Reinmann (2005) justifiably raises the question of whether educational research remains merely an institution that evaluates innovations but does not itself generate innovative approaches to solutions. Ann Brown (1992) was the first scientist who tried to combine theory and practice (Anderson & Shattuck, 2012), because “an effective intervention should be able to migrate from our experimental classroom to average classrooms operated by and for average students and teachers, supported by realistic technological and personal support” (Brown, 1992, p. 143). Over the course of the subsequent years, more and more researchers addressed this focus and DBR was established as a methodological research approach (McKenney & Reeves, 2013). Shavelson and colleagues (2003) coined a generally accepted definition of this particular research method: “Such research, based strongly on prior research and theory and carried out in educational settings, seeks to trace the evolution of learning in complex, messy classrooms and schools, test and build theories of teaching and learning, and produce instructional tools that survive the challenges of everyday practice” (p. 25). Thus, the aim of this approach is to generate innovative solutions to educational problems relevant to practice as well as to draw findings relevant to science (Klees & Tillmann, 2015; Lehmann-Wermser & Konrad, 2016; Schmiedebach & Wegner, 2021; Schäfers & Wegner, 2021a; Weiser, 2020). Neither educational research nor educational practice should be restricted (Lagemann, 2002). Consequently, the gap between research and practice should be reduced (Euler & Sloane, 2014; Mintrop, 2019; Reinmann, 2005). For this purpose, actors from practice who can provide scientifically relevant insights for action- oriented research (Gräsel, 2010; Sandoval & Bell, 2004) are involved in this iterative research process (Hahn et al., 2019), by describing the prevailing conditions in the research field (Möller, Kleickmann & Tröbst, 2009). Design-Based Research Approach in the Project “Kleine BegInNa” As can be seen from the explanations on DBR, the approach mostly includes the school as an educational institution and practical actor. However, education is equally imparted in the pre-school sector (MSB & MKFFI, 2018). In Germany, pre-school education constitutes the first stage of the educational system (Seyda, 2009). Thus, it is of particular relevance to develop innovative approaches for early education. Studies that demonstrate that early promotion of science education has a sustainable impact on children's cognitive abilities (e.g., Anders et al., 2018, Claessens & Engels, 2013; Guo, Piasta & Bowles, 2015, Morgan et al., 2016; Saçkes, 2013), support this conclusion. For this reason, the project "Kleine BegInNa – smalls ones gifted in natural sciences" was founded in 2019 following a DBR approach at “OZHB” (Osthushenrich-Center for Gifted Research at the Faculty of Biology) at “Bielefeld University” (Schäfers, 2023; Schäfers & Wegner, 2021b; Wegner et al., 2020). The focus of this project is on science education and promotion of scientific competences in early 8 Global Education Review 11 (1) childhood (e.g., Schäfers & Wegner, 2020b; 2021c; 2022c). By following the DBR approach, the research objectives of the project were determined. These steps will be focused on in the following. Preliminary Research (Research Question of the Project) A systematic literature review was conducted in the project “Kleine BegInNa” as preliminary research in order to assess the effects of science promotion in early education and to highlight conditions for the successful implementation of such programs (Schäfers & Wegner, 2020a). In this step of DBR, thus, the current situation is identified after firstly defining the problem (Schmiedebach & Wegner, 2021). The analysis showed that children who received science support benefited both cognitively (Lehmann, Rademacher & Müller, 2016; Reichelt, 2014; Steffensky et al., 2012; Windt, 2011) and socially (Lehmann, Rademacher & Müller, 2016; Windt, 2011). This was manifested in higher performance, which they were able to retrieve in the short term and in the long term after the support offers, as well as in their positive behavior during group work. The better the support was adapted to the children's competences, the more positive the effects were (Klemm et al., 2019). However, it was also pointed out that there was no comprehensive scientific talent test that measures the children's prior knowledge in pre- school (Carstensen, Lankes & Steffensky, 2011; Nölke, 2013; Steffensky, Lankes & Carstensen, 2012; Ziegler & Hardy, 2015). Furthermore, the success of the promotion was closely related to the competences of the pedagogical professionals in the daycare centers, who often show a high need for further training in the field of sciences (Bruns, 2014; Fischnaller, 2012; Kauertz & Gierl, 2014; Klemm et al., 2019; Schuler, 2013). Thus, a review of the relevant literature supported the need to develop a scientific talent test as well as in-service training for educational professionals with a focus on science education (Schäfers & Wegner, 2020a). Prototyping Phase (Test Instrument) As a next step, a scientific talent test was developed as a prototype that measures the children's competences relevant to science in the prototype phase (Schäfers & Wegner, 2021b; Schäfers & Wegner, 2022b). For this purpose, the term scientific competence was defined for the context of the project. Additionally, a theoretical basis for the scientific talent test was chosen. Scientific Competences Since there is only a little known so far about the field of science competences in pre- school education (Anders et al., 2018), general explanations of the concept of competence will be given first and then transferred to the field of natural sciences. The term competence describes a person's ability to do something (North, Reinhardt & Sieber-Suter, 2018). Long before the competence-oriented educational plans were introduced in school education in Germany (Künzli, 2010), Chomsky (1973) recognized the relevance of competence acquisition and elevated competence to a key qualification. However, until today scientists do not agree on the exact definition of competence (Erpenbeck & Rosenstiel, 2003). A widely accepted definition is provided by Weinert (2001), who defines competence as the cognitive abilities and skills that individuals Quality of Research Based on Design-Based Research Approach 9 possess or can learn in order to develop problem-solving strategies. Based on the American Association for the Advancement of Science (AAAS; 2009) and the PISA Consortium Germany (2007), the National Educational Panel Study (NEPS) has combined this definition with the concept of scientific literacy to obtain a specific definition of competence in the field of science (Hahn & Schöps, 2019). According to their approach, scientific competence is composed of content-related (content areas: substances, development, interaction and systems) and process-related (nature of science and process for generating scientific knowledge) components, which collectively form the foundation for building scientific competence (Hahn et al., 2013). Additionally, the IPN - Leibniz Institute for Science and Mathematics Education has already established a definition of scientific competence based on the PISA Consortium Germany (2007; IPN, n.d.). In terms of the scientific-literacy approach, scientific competence is divided into three sub- competences that lead to the acquisition of scientific ways of thinking and working: • Identifying and formulating questions that can be investigated and answered scientifically, • Describing and explaining scientific phenomena, and • Interpreting scientific evidence (IPN, n.d.; Schäfers & Wegner, 2022b). However, the focus is not only on scientific knowledge, but also on the nature of science and the promotion of positive affective characteristics, such as the emotional experience of, curiosity about, and pleasure in scientific phenomena, as well as the development of a sense of responsibility for the use of knowledge. For early childhood education, Steffensky (2017) differentiated the scientific ways of thinking and working into smaller sub-competences, which also form sub-competences of the scientific process of gaining knowledge: Asking questions; Hypothesizing; Observing; Measuring; Planning and conducting investigations; Comparing, arranging and classifying; Analyzing, interpreting, concluding and generalizing data; Arguing; Using models; Documenting. These competences need to be promoted at a lower level in pre-school already. Thus, a scientific talent test for pre-school should assess these competences (Schäfers & Wegner, 2022b). CHC-Theory of Cognitive Abilities as a Model for the Scientific Talent Test To cluster such competences which are relevant for science, the CHC-theory of cognitive abilities was used as a theoretical basis (Schäfers & Wegner, 2021b). The CHC-theory is a theory synthesis by Cattell (1963), Horn (Horn, 1991; Horn & Blankson, 2005), and Carroll (1993), taking intelligence as a hierarchical construct and subdividing intelligence into different general and specific abilities (Flanagan & Dixon, 2013; Mickley & Renner, 2019). This theory was chosen on the one hand because it is a highly accepted scientific theory (Baudson, 2012) and on the other hand because there is a strong connection between intelligence and competence (Preckel & Holling, 2006). Thus, the competences according to Steffensky (2017) were classified within the intelligence model, so that the following general abilities were identified for the scientific talent test (Schäfers & Wegner, 2022b): • Fluid intelligence • Quantitative knowledge • Visual processing • Long-term retrieval • Processing speed 10 Global Education Review 11 (1) Scientific Talent Test The scientific talent test is a single test in which the above-mentioned ability areas that are relevant for scientific talent are measured in seven subtests. Due to the age of the children, the test is not a paper-pencil test, but rather a series of riddles, experiments, and hands-on experiences in which the children's answers are recorded and evaluated on a protocol sheet. The test sessions of the children are conducted by researchers of the project “Kleine BegIn-Na”, who were extensively trained in advance. Further information with more detailed in-sights into the test instrument can be found in other publications of the project (Schäfers & Wegner, 2022a; Schäfers & Wegner, 2022b; Schäfers et al., 2023). Assessment Phase (Study Design) According to the DBR approach, the first two phases were followed by the assessment phase in the project “Kleine BegInNa”, in which the scientific talent test was piloted in several research cycles from 2019 to 2022. The first version of the test instrument was initially piloted with a small sample (N = 7) in July 2019, as it is appropriate to conduct the first evaluation and revision based on the results of a small group (Scheersoi & Tessartz, 2019). Based on the evaluation of the testing, experiences during the testing, and feedback from pedagogical professionals, changes and revisions were made until early 2020, before the adapted instrument was piloted in a larger study. During this cycle, it became apparent that, in addition to the quantitative recording of children's cognitive performance, a more qualitative investigation of the children’s behavior was also relevant for interpreting the results. Thus, after the second cycle, an observation sheet was developed to be used during the test (Schäfers et al., 2020). The combination of a scientific test and observation sheets was used in the third research cycle. In total, data from N = 247 children are available. At the same time, the scientific talent test was already tested for the classical quality criteria of quantitative research with partial samples (Schäfers & Wegner, 2022a). In addition, the internal structure has already been determined by exploratory factor analysis (Schäfers et al., 2023) with a small sample. In order to make a valid assumption about the internal structure, the exploratory factor analysis has to be repeated with the whole sample. Therefore, the next research cycle is presented below, which focused on the question: To what extent does exploratory factor analysis reveal an internal structure within the scientific talent test? Solution (Results) As a solution, an exploratory factor analysis was conducted involving the complete data set in order to determine the internal structure of the scientific talent test and thus to determine what the test instrument measures. The requirements for conducting an exploratory factor analysis were tested by Bartlett's test and the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO). The Bartlett’s test confirms whether the constructs correlate with each other or not (Backhaus et al., 2021). The KMO indicates whether the data are suitable for exploratory factor analysis, with cut-off values set by Kaiser and Rice (1974) and considered performable above a value of 0.5. Both Bartlett’s test (chi-square (21) = 380.498, p < .001) and the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO = .816; classified as good) showed that an exploratory factor analysis can be performed with the variables included. Quality of Research Based on Design-Based Research Approach 11 Following the suitability check, a principal component analysis with direct-oblimin-rotation was performed. This method was chosen because the included factors correlated with each other (Field, 2018). As can be seen from the screeplot figure (see figure 1), the principal component analysis identified one factor that has an eigenvalue greater than 1.0. The one- factor solution could explain 43.56% of the total variance. Figure 1. Screeplot of principal component analysis (SPSS-output). The individual dimensions included in the analysis contribute equally to the explanation of the total variance. Furthermore, the component matrix showed that the individual components could all be clearly assigned to factor 1, since the loadings of the components on factor 1 were at least 0.609. The communalities as well as the factor loadings can be seen in the combined table 1 (see Appendix). Discussion Since only one factor had a higher eigenvalue than 1.0 and in the screeplot clearly only one factor was above the bend, it can be assumed that only one factor was explained by the scientific talent test. This factor may be the “scientific talent” of the children. The individual components of the scientific talent test, which were the basis for the test development, could apparently not be identified by the test instrument. Thus, children who perform well on one subtest are likely to perform well on the other subtests. Conversely, it can be assumed that children who perform less well on one subtest will also perform lower on the other subtests. This indicates that all subtests seem to contribute to the description of scientific talent without differentiating between individual scientific competences. This in turn can be explained not least by the fact that the child's brain only develops over the course of primary school and that individual ability areas can rarely be distinguished from one another before the age of six (Büttner, 2017). However, it should be noted that the screeplot can only be interpreted meaningfully when the sample size is greater than N = 300. Thus, the results of the analysis with the existing sample of the study are a tendency and a clear indication, but the result should also be checked for larger sample sizes. Furthermore, what remains debatable about the study is that no demographic data except the age of the children were included in the analysis of the results. On the one hand, this increases comparability, but on the other hand, it does not account for differences in performance due to language problems or a low ability to concentrate. Overall, as a solution in the DBR approach, the scientific talent test can be accepted as a valid test instrument. However, this is not the final step in the process of DBR. Since it is an iterative and cyclical process, this instrument can be used in further research cycles with stakeholders from research and 12 Global Education Review 11 (1) practice and can be investigated for different target groups. In addition, the scientific talent test can also create another field of research by integrating the test instrument into training programs for educational professionals. In this process, the training offers can also be evaluated. The constant revision and repeated implementation then correspond to the DBR approach. Discussion of Quality of Research Based on Design-Based Research Approach As can be deduced from the previous chapters, the DBR approach pursues a twofold goal: on the one hand, the highest possible benefit for practice should be derived from the designed interventions, and on the other hand, the interventions should generate theoretical insights for science (Reinmann, 2022). Thus, the quality of implementation must be determined on both levels. However, there are currently hardly any commonly accepted standards for the DBR approach to assess the quality of DBR research (Bakker, 2018; Hoadley, 2004; Tulodziecki, Herzig & Grafe, 2014). Standards are defined as scientific claims that ensure or improve the quality of research (Reinmann, 2022). In contrast, quality criteria are the operationalized tools to measure quality and survey compliance (Gerhold et al., 2015). Objectivity, reliability and validity are often referred to as 'classical' quality criteria of scientific research, which is certainly plausible, since research should always be unbiased and factual, as well as dependable and binding (Reinmann, 2022). However, the three quality criteria are characteristics that are strongly related to ideal empirical research in the natural sciences (Bortz & Döring, 2006) and thus can rarely serve as standards for questions in educational science (Reinmann, 2022). Beyond these three criteria, there are several other standards for scientific quality that can be applied to research, such as appropriateness, replicability, or transferability (just to name a few; Reinmann, 2022). However, even these are difficult to apply because they cannot be clearly separated and there is no agreement on how to verify these criteria (ibid.). Thus, for DBR research in particular, the question arises: What standards can be applied in DBR to determine the quality of the research approach and design? For this purpose, Reinmann (2022), in her debate about standards in DBR research and the role of design in this tension, establishes two levels on which standards in DBR research should be based. As the term DBR suggests, she distinguishes between the levels of scientificity and design appropriateness to do justice to the focus of research and practice and to reflect the comprehensive character of the DBR approach. Table 2 (see Appendix) presents an overview of the suggested quality criteria for DBR research according to Reinmann (2022) and gives a brief definition for each of them. By focusing both science and design aspects, the evaluation of the quality of research in the sense of DBR can be successful. Even if the comparison of individual criteria (such as systematicity and openness or generalizability and context sensitivity) creates certain tensions, balancing these tensions constitutes one of the benefits of DBR (Reinmann, 2022). Conclusion and Outlook In conclusion, the research principle of DBR does not constitute a panacea or an all- purpose solution for all problems in educational Quality of Research Based on Design-Based Research Approach 13 research (Schmiedebach & Wegner, 2021), but the strong link between theory and practice as well as the inclusion of many actors involved in the process can create a higher degree of understanding for the research field and thus also stimulate the development of innovative solutions. This is demonstrated not least by the introduced project “Kleine BegInNa”, which follows a DBR approach with its methodological orientation (Schäfers & Wegner, 2021b). However, just as DBR is more than just the addition of research and practice, it is also complex in terms of assessing the quality of DBR research. With her list of possible quality criteria for this research approach, Reinmann (2022) has shown how complex DBR needs to be thought of and that it is more than just applying the classical quality criteria of empirical social research. For this reason, the presented project must be evaluated by using the quality criteria of DBR. This also entails potential revisions and further development regarding its quality in further research cycles. Acknowledgment We would like to thank Carolin Zehne for the linguistic revision. References AAAS (American Association for the Advancement of Science; 2009). Benchmarks for science literacy. Project 206. http://www.project2061.org/publication s/bsl/online/index.php Anders, Y., Barenthien, J., Hardy, I., Hartinger, A., Kästner, R., Leuchter, M., Oppermann, E., Pauen, S., Rank, A., Steffensky, M., Taskinen, P., Tietze, S., Wildemann, A., & Ziegle, T. (2018). Wirkungen naturwissenschaftlicher Bildungsangebote auf pädagogische Fachkräfte und Kinder. Opladen / Berlin / Toronto: Verlag Barbara Budrich. https://doi.org/10.3224/84742274 Anderson, T., & Shattuck, J. (2012). Design- Based Research: A Decade of Progress in Education Research? Educational Researcher, 41(1), 16-25. https://doi.org/10.3102/0013189X1142 8813 Asbrand, B. (2009). Schule verändern, Innovationen implementieren. Zeitschrift für internationale Bildungsforschung und Entwicklungspädagogik (ZEP), 32(1), 15-21. Backhaus, K., Erichson, B., Gensler, S., Weiber, R., & Weiber, T. (2021). Multivariate Analysis. An Application-Oriented Introduction. Wiesbaden: Springer Gabler. https://doi.org/10.1007/978-3- 658-32589-3 Bakker, A. (2018). Design research in education. A practical guide for early career researcher. New York: Routledge. Baudson, G. (2012). Der Aufbau der Intelligenz. Die CHC-Theorie als Strukturmodell kognitiver Fähigkeiten. MinD-Magazin, 91, 8-10. Bortz, J., & Döring, N. (2006). Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler. 14 Global Education Review 11 (1) Berlin: Springer. https://doi.org/10.1007/978-3-540- 33306-7 Brown, A. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178. https://doi.org/10.1207/s15327809jls02 02_2 Bruns, J. (2014). Adaptive Förderung in der elementarpädagogischen Praxis: eine empirische Studie zum didaktischen Handeln von Erzieherinnen und Erziehern im Bereich Mathematik. Münster / New York: Waxmann. Büttner, G. (2017). Kognitive Entwicklung und Förderung im Grundschulalter. In B. Kracke & P. Noack (Eds.), Handbuch Entwicklungs- und Erziehungspsychologie (pp. 1-27). Berlin / Heidelberg: Springer VS. https://doi.org/10.1007/978-3-642- 54061-5_8-1 Carroll, J. (1993). Human cognitive abilities: a survey of factor-analytic studies. Cambridge: Cambridge University Press. https://psycnet.apa.org/doi/10.1017/CB O9780511571312 Carstensen, C. H., Lankes, E.-M., & Steffensky, M. (2011). Ein Modell zur Erfassung naturwissenschaftlicher Kompetenz im Kindergarten. Zeitschrift für Erziehungswissenschaft, 14, 651-669. https://doi.org/10.1007/s11618-011- 0240-1 Cattell, R. (1963). Theory of fluid and crystallized intelligence: a critical experiment. Journal of Educational Psychology, 54, 1-22. https://psycnet.apa.org/doi/10.1037/h0 046743 Chomsky, N. (1973). Aspekte der Syntax- Theorie. Frankfurt a.M.: Suhrkamp. Claessens, A., & Engels, M. (2013). How Important Is Where You Start? Early Mathematics Knowledge and Later School Success. Teacher College Record, 115(6), 1-29. Dehmel, A. (2018). Transfer im Fokus: Einblicke in aktuelle Entwicklungen in Deutschland. Transfer Forschung Schule, 4(4), 142-148. Deutsches PISA-Konsortium (2007). PISA 2006. Die Ergebnisse der dritten internationalen Vergleichsstudie. Münster: Waxmann. Erpenbeck, J., & Rosenstiel, L.v. (2003) (Eds.). Handbuch Kompetenzmessung. Stuttgart: Schäffer-Poeschel. Euler, D., & Sloane, P. (2014). Editorial. Zeitschrift für Berufs- und Wirtschaftspädagogik – Beihefte, 27, 7- 12. Field, A. (2018). Discovering statistics using IBM SPSS statistics. London: SAGE Publications Ltd. Quality of Research Based on Design-Based Research Approach 15 Fischnaller, S. (2012). Zum Stand mathematischer Lernumgebungen in Kindergärten des Kindergartensprengels Brixen. Laureate thesis. Bozen. Flanagan, D., & Dixon, S. (2013). The Cattell- Horn-Carroll Theory of Cognitive Abilities. In C. Reynolds, K. Vannest, & E. Fletcher-Janzen (Eds.), Encyclopedia of Special Education. Hoboken: John Wiley & Sons Inc. Gerhold, L., Holtmannspötter, D., Neuhaus, C. Schüll, E., Schulz-Montag, B., Steinmüller, K., & Zweck, A. (2015). Einleitung. In L. Gerhold, D. Holtmannspötter, C. Neuhaus, E. Schüll, B. Schulz-Montag, K. Steinmüller & A. Zweck (Eds.), Standards und Gütekriterien der Zukunftsforschung (pp. 9-15). Wiesbaden: Springer VS. https://doi.org/10.1007/978-3-658- 07363-3_1 Gräsel, C. (2010). Stichwort: Transfer und Transferforschung im Bildungsbereich. Zeitschrift für Erziehungswissenschaften, 13, 7-20. https://doi.org/10.1007/s11618-010- 0109-8 Gundlach, E. (2003). Nach dem PISA-Schock: Höhere Bildungsaufgaben oder umfassende Bildungsreform? In T. Hansel (Eds.), Pisa – und die Folgen? Die Wirkung von Leistungsvergleichsstudien in der Schule (pp. 216-236). Herbolzheim: Centaurus Verlag & Media. Guo, Y., Piasta, S., & Bowles, R. (2015). Exploring Preschool Children’s Science Content Knowledge. Early Education and Development, 26(1), 125-146. https://doi.org/10.1080/10409289.201 5.968240 Hahn, I., & Schöps, K. (2019). Bildungsunterschiede von Anfang an? Die Bedeutung von Struktur- und Prozessmerkmalen für die naturwissenschaftliche Kompetenz von Vorschulkindern mit und ohne Migrationshintergrund. Frühe Bildung, 8(1), 3-12. https://doi.org/10.1026/2191- 9186/a000405 Hahn, I., Schöps, K., Rönnebeck, S., Martensen, M., Hansen, S., Saß, S., Dalehefte, I., & Prenzel, M. (2013). Assessing scientific literacy over the lifespan - A description of the NEPS science framework and the test development. Journal for Educational Research Online (JERO), 5(2), 110-138. https://doi.org/10.25656/01:8427 Hahn, S., Klewin, G., Koch, B., Kuhnen, S.U., Palowski, M., & Stiller, C. (2019). Über Praxis-forschung zum Transfer von Innovationen? In C. Schreiner, C.Wiesner, S. Breit, P. Dobbelstein, M. Heinrich & U. Steffens (Eds.), Praxistransfer Schul- und Unterrichtsentwicklung (pp. 141-152). Münster / New York: Waxmann. Hoadley, C. M. (2004). Methodological alignment in design-based research. 16 Global Education Review 11 (1) Educational Psychologist, 39(4), 203- 212. https://doi.org/10.1207/s15326985ep39 04_2 Horn, J. (1991). Measurement of intellectual capabilities: a review of theory. In K. McGrew, J. Werder & R. Woodcock (Eds.), Technical Manual. Woodcock- Johnson Psycho-Educational Battery- Revised (pp. 197-232). Chicago: Riverside. Horn, J., & Blankson, A. (2005). Foundations for better understanding of cognitive abilities. In D. Flanagan & P. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 73-98). Guilford: The Guilford Press. IPN (n.d.). Bereich naturwissenschaftliche Kompetenz. https://archiv.ipn.uni- kiel.de/PISA/index.html Juuti, K., & Lavonen, J. (2006). Design-Based Research in Science Education: One Step Towards Methodology. Nordic Studies in Science Education (NorDiNa), 2(2), 54-68. https://doi.org/10.5617/nordina.424 Kaiser, H. F., & Rice, J. (1974). Little Jiffy, Mark IV. Educational and Psychological Measurement, 34, 11-117. https://doi.org/10.1177/0013164474034 00115 Kauertz, A., & Gierl, K. (2014). Naturwissenschaften im Elementarbereich. In D. Kucharz, K. Mackowiak, S. Ziroli, A. Kauertz, E. Rathgeb-Schnierer & M. Dieck (Eds.), Professionelles Handeln im Elementarbereich (PRIMEL). Eine deutsch-schweizerische Videostudie (pp. 167-178). Münster / New York: Waxmann. Klees, G., & Tillmann, A. (2015). Design-Based Research als Forschungsansatz in der Fachdidaktik Biologie. Journal für Didaktik der Biowissenschaften, 6(2015), 91-110. Klemm, J., Kohlhauf, L., Boone, W., Sodian, B., & Neuhaus, B. (2019). Förderung biologischer Beobachtungskompetenz im Kindergarten. Frühe Bildung, 8(1), 22-29. https://doi.org/10.1026/2191- 9186/a000407 Künzli, R. (2010). Lehrpläne, Bildungsstandards und Kompetenzmodelle. Eine problematische Vermischung von Funktionen. Beiträge zur Lehrerbildung, 28(3), 440-452. https://doi.org/10.25656/01:13761 Lagemann, E. (2002). An Elusive Science: The Troubling History of Education Research. Chicago: University of Chicago Press. Lehmann, W., Rademacher, J., & Müller, I. (2016). Zu den Effekten eines mathematischen Förderprogramms: „Früh über sich, ... ‒ gewusst wie!“ In A. Schmitt, A. Schwentesius & E. Sterdt (Eds.), Neue Wege für frühe Bildung und Förderung im Forschungsfeld Quality of Research Based on Design-Based Research Approach 17 Mathematik, Informatik, Naturwissenschaften und Technik (MINT)(pp. 30-44). Baltmannsweiler: Schneider Hohengehren Verlag. Lehmann-Wermser, A., & Konrad, U. (2016). Design-Based Research als eine der Praxis verpflichtete, theoretisch fundierte Methode der Unterrichtsforschung und -entwicklung. Methodologische Grundlagen, dargestellt am Beispiel eines Forschungsprojekts im Bandklassen- Unterricht. In J. Knigge, & A. Niessen (Eds.), Musikpädagogik und Erziehungswissenschaft (pp. 265-280). Münster & New York: Waxmann. McKenney, S., & Reeves, T. (2013). Systematic Review of Design-Based Research Progress: Is a Little Knowledge a Dangerous Thing? Educational Researcher, 42(2), 97-100. https://doi.org/10.3102/0013189X1246 3781 Mickley, M., & Renner, G. (2019). Auswahl, Anwendung und Interpretation deutschsprachiger Intelligenztests für Kinder und Jugendliche auf Grundlage der CHC-Theorie: Update, Er-weiterung und kritische Bewertung. Praxis der Kinderpsychologie und Kinderpsychiatrie, 68(4), 323-343. https://doi.org/10.13109/prkk.2019.68. 4.323 Mintrop, R. (2019). Designbasierte Schulentwicklung – ein kurzer Abriss. In C. Schreiner, C.Wiesner, S. Breit, P. Dobbelstein, M. Heinrich, & U. Steffens (Eds.), Praxistransfer Schul- und Unterrichtsentwicklung (pp. 35-48). Münster / New York: Waxmann. Möller, K., Kleickmann, T., & Tröbst, S. (2009). Die forschungsgeleitete Entwicklung von Unterrichtsmaterialien für die frühe naturwissenschaftliche Bildung. Beiträge zur Lehrerbildung, 27(3), 415- 423. https://doi.org/10.25656/01:13711 Morgan, P., Farkas, G., Hillemeier, M., & Maczuga, S. (2016). Science Achievement Gaps Begin Very Early, Persist, and Are Largely Explained by Modifiable Factors. Educational Researcher, 45(1), 18-35. https://doi.org/10.3102/0013189X1663 3182 MSB, & MKFFI (2018). Bildungsgrundsätze. Mehr Chancen durch Bildung von Anfang an. Freiburg: Herder. Nölke, C. (2013). Erfassung und Entwicklung des naturwissenschaftlichen Interesses von Vor-schulkindern. Dissertation. Kiel: Christian-Albrechts-Universität zu Kiel. North, K., Reinhardt, K., & Sieber-Suter, B. (2018). Kompetenzmanagement in der Praxis. Mitarbeiterkompetenzen systematisch identifizieren, nutzen und entwickeln. Wiesbaden: Springer Gabler. https://doi.org/10.1007/978-3- 8349-3696-7 Preckel, F., & Holling, H. (2006). Die Rolle von Intelligenz und Begabung für 18 Global Education Review 11 (1) Handlungskompetenz: am Beispiel beruflicher Hochbegabung. Bildung und Erziehung, 59(2), 167-178. https://doi.org/10.7788/bue.2006.59.2. 167 Reichelt, J. (2014). Vorschulische Förderung mathematischer Kompetenzen. Dissertation. Bielefeld: Universität Bielefeld. Reinmann, G. (2005). Innovation ohne Forschung? Ein Plädoyer für Design- Based Research-Ansatz in der Lehr- Lernforschung. Unterrichtswissenschaft, 33(1), 52-69. https://doi.org/10.25656/01:5787 Reinmann, G. (2022). Was macht Design-Based Research zu Forschung? Die Debatte um Standards und die vernachlässigte Rolle des Designs. EDeR – Educational Design Research, 6(2), 1-22. https://doi.org/10.15460/eder.6.2.1909 Saçkes, M. (2013). Children’s Competencies in Process Skills in Kindergarten and Their Impact on Academic Achievement in Third Grade. Early Education and Development, 24(5), 704-720. https://doi.org/10.1080/10409289.201 2.715571 Sandoval, W., & Bell, P. (2004). Design-Based Research Methods for Studying Learning in Context: Introduction. Educational Psychologist, 39(4), 199- 201. https://doi.org/10.1207/s15326985ep39 04_1 Schäfers, M.S. (2023). Entwicklung und Pilotierung eines naturwissenschaftlichen Begabungstests für den Elementarbereich. Dissertation: Universität Bielefeld. Schäfers, M.S., Höhne, M., Rehkemper, J., & Wegner, C. (2020). Darstellung eines Mixed-Methods-Ansatzes zur Bestimmung naturwissenschaftlicher Begabung im Kindergartenalter. Diskurs Kindheits- und Jugendforschung, 15(4), 461-466. https://doi.org/10.3224/diskurs.v15i4.1 0 Schäfers, M.S., Perleth, C., Bueno Castellanos, C.P., Lübcke, H., & Wegner, C. (2023). How can scientific talent be recognized in the early years? Validating a scientific talent test for pre-school age. Journal of Gifted Education and Creativity, 10(1), 33-43. Schäfers, M.S., & Wegner, C. (2020a). Diagnose und Förderung von naturwissenschaftlicher Begabung in der Kita. Darstellung des aktuellen Forschungsstands. Diskurs Kindheits- und Jugendforschung, 15(1), 70-86. https://doi.org/10.3224/diskurs.v15i1.0 6 Schäfers, M. S., & Wegner, C. (2020b). Facettenreich. Die Augen der Fliege kennenlernen. Entdeckungskiste, 2020(6), 58-59. Schäfers, M. S., & Wegner, C. (2021a). Design- Based Research in der Quality of Research Based on Design-Based Research Approach 19 Hochbegabungsforschung. Ein multiperspektivischer Problemlöser?! Labyrinth, 44(2), 30-31. Schäfers, M. S., & Wegner, C. (2021b). „Kleine BegInNa“: Ein Projekt zur Bestimmung und Förderung naturwissenschaftlicher Kompetenzen von Kitakindern. ElFo - Elementarpädagogische Forschungsbeiträge, 3(1), 9-20. https://doi.org/10.25364/18.3:2021.1.1 Schäfers, M. S., & Wegner, C. (2021c). Feuer im Fokus: Experimente für die Kita. Klein & groß: mein Kita-Magazin, 2021(2-3), 35-37. Schäfers, M. S., & Wegner, C. (2022a). Quality Criteria Check – Testing of a Scientific Giftedness Instrument for Preschool Age. Journal of Innovation in Psychology, Education and Didactics (JIPED), 26(1), 25-38. https://doi.org/10.29081/JIPED.2022.2 6.1.03 Schäfers, M.S., & Wegner, C. (2022b). Originalarbeit: Die naturwissenschaftlichen Fähigkeiten von Kindern entdecken. Vorstellung eines naturwissenschaftlichen Begabungstests für den Elementarbereich. Frühförderung interdisziplinär, 41(1), 18-31. http://dx.doi.org/10.2378/fi2022.art02 d Schäfers, M. S., & Wegner, C. (2022c). Das Flaschen-Thermometer. Pädagogisches Material selbst hergestellt. Kindergarten heute: Fachzeitschrift für Erziehung, Bildung und Betreuung von Kindern, 52(3), 36-37. Scheersoi, A., & Tessartz, A. (2019). Design- based Research – ganz praktisch! Bildungsforschung, 17(1), 1-7. https://doi.org/10.25539/bildungsforsc hun.v0i1.283 Schmiedebach, M., & Wegner, C. (2021). Design- Based Research als Ansatz zur Lösung praxisrelevanter Probleme in der fachdidaktischen Forschung. Bildungsforschung, 19(2), 1-10. https://doi.org/10.25539/bildungsforsc hun.v0i2.413 Schuler, S. (2013). Mathematische Bildung im Kindergarten in formal offenen Situationen. Eine Untersuchung am Beispiel von Spielen zum Erwerb des Zahlbegriffs. Münster / New York / München / Berlin: Waxmann. Seyda, S. (2009). Kindergartenbesuch und späterer Bildungserfolg. Zeitschrift für Erziehungs-wissenschaft, 12(2), 233- 251. https://doi.org/10.1007/s11618- 009-0073-3 Shavelson, R.J., Phillips, D.C., Towne, L., & Feuer, M.J. (2003). On the science of education design studies. Educational Researcher, 32(1), 25-28. https://doi.org/10.3102/0013189X0320 01025 Steffensky, M. (2017). Naturwissenschaftliche Bildung in Kindertageseinrichtungen. Weiterbildungsinitiative 20 Global Education Review 11 (1) Frühpädagogische Fachkräfte, WiFF Expertisen, Volume 48. München. Steffensky, M., Lankes, E.-M., & Carstensen, C. (2012). Was bedeutet naturwissenschaftliche Kompetenz bei Fünfjährigen und wie kann man sie erfassen. In M. Gläser-Zikuda, T. Seidel, C. Rohlfs, A. Gröschner & S. Ziegelbauer (Eds.), Mixed Methods in der empirischen Bildungsforschung (pp. 107-120). Münster / New York / München / Berlin: Waxmann. Steffensky, M., Lankes, E.-M., Carstensen, C., & Nölke, C. (2012). Alltagssituationen und Experimente; Was sind geeignete naturwissenschaftliche Lerngelegenheiten für Kindergartenkinder? Zeitschrift für Erziehungswissenschaften, 15, 37-54. https://doi.org/10.1007/s11618-012- 0262-3 Tulodziecki, G., Herzig, B., & Grafe, S. (2014). Medienpädagogische Forschung als gestaltungsorientierte Bildungsforschung vor dem Hintergrund praxis- und theorierelevanter Forschungsansätze in der Erziehungswissenschaft. Medienpädagogik. Zeitschrift für Theorie und Praxis in der Medienbildung, 2014 (Occasional Papers), 1-18. https://doi.org/10.21240/mpaed/00/20 14.03.10.X Wegner, C., Zehne, C., Wiese, I., & Schäfers, M.S. (2020). Introducing a Concept for Supporting Scientifically Gifted Students - the Osthushenrich-Center for Giftedness Research (OZHB) at the Department of Biology at Bielefeld University, Germany. Journal of Gifted Education and Creativity, 7(1), 13-19. Weinert, F. (2001). Leistungsmessungen in Schulen. Weinheim / Basel: Beltz. Weiser, L. (2020). Interesse an der Natur bei Kindergarten- und Grundschulkindern - Studien zur Gestaltung und Wirkung des Forschenden Lernens an außerschulischen Lernorten. Dissertation: Rheinische Friedrich- Wilhelms-Universität zu Bonn. Windt, A. (2011). Naturwissenschaftliches Experimentieren im Elementarbereich. Evaluation verschiedener Lernsituationen. Dissertation. Dortmund: Technische Universität Dortmund. Ziegler, T., & Hardy, I. (2015). Die Erfassung naturwissenschaftlicher Kompetenz im Vorschulalter. Ergebnisse einer Pilotierungsstudie. In K. Liebers, B. Landwehr, A. Marquardt & K. Schlotter (Eds.), Lernprozessbegleitung und adaptives Lernen in der Grundschule (pp. 211-216). Wiesbaden: Springer Fachmedien. https://doi.org/10.1007/978-3-658- 11346-9_27 Quality of Research Based on Design-Based Research Approach 21 Appendix Table 1. Communalities for the elucidation of the total variance and component matrix. Communalities for elucidation of total variance Component matrix Initially Extraction Component 1 Subtest 1 1.000 .357 .613 Subtest 2 1.000 .458 .677 Subtest 3 1.000 .521 .722 Subtest 4 1.000 .374 .611 Subtest 5 1.000 .553 .743 Subtest 6 1.000 .371 .609 Subtest 7 1.000 .397 .630 Notes: Due to the single-factor solution, no rotation is necessary. Table 2. Possible standards of DBR research according to Reinmann (2022). Standards at the level of… … Scientificity … Design Appropriateness Systematicity DBR activities should be more systematic in both theory and design than activities conducted in educational practice without research. In addition, knowledge derived from research and design should be more systematically represented by the DBR approach than knowledge acquired in action practices without research (cf. p. 7-8). Future relevance Through DBR, interventions should be created and implemented to design future educational offers and to adapt empirical research and theoretical approaches to them. (cf. p. 12). Perspectivity According to the DBR approach, there is often a high level of involvement of the researchers. This involvement in the research process should thus be consciously handled and reflected (cf. p. 8-9). Openness In DBR, being open to unexpected results is of great importance, as the research process might lead away from initial goals and questions. This can then be deliberately integrated into the research cycles. However, strategies are needed to handle the dynamic course of research (cf. p. 12). 22 Global Education Review 11 (1) Incompleteness Research and science produce new knowledge, yet this knowledge is not fixed, absolute knowledge, but rather constitutes assumptions. Thus, for DBR, it is important that research is recognized as an unfinished process that tends to produce prospective knowledge (cf. p. 9). Context Sensitivity For DBR, one of the main foci is the context, as this is relevant for designing and implementing the interventions. Thus, the theory, empiricism as well as the design activities should be adapted to the context as well as to the actors in a particular setting (cf. p. 12- 13). Generalizability Through research, conclusions should be made that are not only valid for the investigated sample, but that are also transferable to other groups and contexts. Therefore, DBR should produce results that can be generalized. Furthermore, it should be shown how these findings can be applied under which conditions in other situations (cf. p. 9). Saturation The iterative-cyclical research process is a characteristic aspect of the DBR approach. However, it should be continuously observed when a cycle in theory, empiricism and design has been finished for the moment in order to connect further cycles or to determine the next steps in the research process (cf. p. 13). Transparency To ensure traceability and transparency in DBR, decisions regarding theoretical and empirical aspects as well as design should be made transparent. This also includes a coherent justification of key aspects without getting lost in details (cf. p. 10). Knowledge Diversity Different forms of knowledge should be implemented in the DBR process: research should involve different theoretical, empirical, and design sources that integrate different types of knowledge. These should be related to each other (cf. p. 13). Publicity Because DBR findings are relevant to practice and research, the findings should be made publicly available in appropriate forums (or firstly establishing such forums and opportunities for publications), taking the intended audiences into account (cf. p. 10). Normativity Decisions regarding the values and the target should be actively included and justified in the DBR research process. If something changes during the research process, it can be modified and adjusted (cf. p. 14).