International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol 17 No 10 (2023) Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach Designing of a Flipped STEM Classroom Engineering- Based Module: Fuzzy Delphi Approach https://doi.org/10.3991/ijim.v17i10.38217 Khadijeh Naser1(), Sherin Alamassi1, Zuhrieh Shana1, Jihan Yousef1, Siti Hajar Halili2 1 Al Ain University, Al Ain, United Arab of Emirate 2 University of Malaya, Kuala Lumpur, Malaysia Khadijehmohammad78@gmail.com Abstract—The National Innovation System found that research in Jordan did not contribute sufficiently to economic growth and to solving real-world challenges, particularly those related to STEM (science, technology, engineering, and mathematics). This was attributed to several reasons, including the inability of educational curricula to guide students in dealing with real-world issues. In addition, education in Jordan is content-intensive and primary school children lack the time to build functional competencies such as problem-solving. Therefore, it was necessary to adopt an approach allowing students to learn the theoretical content on their own while allocating class time to practice problem- solving activities with their teacher and peers. To address this gap, this study aimed at designing an engineering-based module for a flipped STEM classroom to aid grade seven students in developing their problem-solving abilities based on Merrill’s first principles of instruction as an instructional design model. To generate the module components and aspects, semi-structured interviews were performed with 9 subject matter experts. Then, 29 experts responded and offered a consensus on what was reached in the previous interview. A total of 36 items were discussed by the expert panel using the Likert 7-point scale in the fuzzy Delphi approach. The five-module aspects namely; the form of STEM integration, the assessment, the resources, and the pre-class and in-class instructional activities, were determined. This research will usher in a new era for the Ministry of Education. in planning and teaching integrated STEM disciplines in line with Jordan’s vision 2025 to equip all learners with 21st- century skills such as problem-solving to enhance education standards to international levels. Keywords—problem-solving skills, Fuzzy Delphi method, Merrill’s first principles of instruction, primary students, flipped classroom 1 Introduction Problem-solving skills are viewed as a global indicator of an effective and successful educational system. Implementing problem-solving strategy enhances student learning, and the understanding of more ideas and in the process elevating the quality of 4 http://www.i-jim.org https://doi.org/10.3991/ijim.v17i10.38217 mailto:Khadijehmohammad78@gmail.com Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach education by engaging students with enjoyable tasks that stimulate thought and deep knowledge. It also strengthens the skills behind cognition and self-learning [1] Nevertheless, mastering problem-solving is a challenging task that may be out of reach for some. Jordanian students have shown poor performance on international scientific and mathematics tests, for example the TIMSS (Trends in International Mathematics and Science Studies); they performed worst when asked to solve complicated problems and integrate new knowledge. One of the reasons students struggle to contend with real challenges is their inability to determine the optimal solution plan [2]. Because existing curricula rarely present students with complex problems to solve, the problems provided are frequently well-structured and do not reflect the complexities they encounter on a daily basis. Global educational systems. however, are racing to improve education outcomes, particularly in STEM education, which has recently demonstrated its close relationship to the economy and solving real-world problems. Another reason is the instructional method. Teachers were discovered to have pedagogical content knowledge (PCK) deficiencies in STEM [3]. They appear to place a greater emphasis on memorisation of scientific concepts and facts than on problem- solving exercises. When presenting problems on a difficult topic such as STEM, teachers should provide examples. For example, STEM integration is a conceptual and intellectual subject, in which students can understand the problem-solving process by using actual engineering experiences to reformulate science and math concepts. During this time, students plan and build to connect STEM topics and achieve “conceptual coherence” [4]. As a potential strategy for integrating STEM disciplines, modeling through engineering design has recently gained international attention. Engineering is interdisciplinary and requires knowledge of science and mathematics to solve problems faced by engineers (learners) [5] while technology integration with mathematics and science enables students to effectively manage their learning tools during learning [6]. Consequently, students are able to address challenges, as real-world scenarios are complex and do not fall under a single domain. In the engineering design process, the teacher acts as a coach and source of feedback so that the students can apply what they have learned in everyday life [7]. This is possible through student-student and student-teacher discussions. Among the many challenges teachers face, however, is how to teach integrated STEM, especially how to integrate engineering and science effectively [8]. Several studies presented the integration approaches [9] yet they fail to provide instructional guidelines for integrating STEM teaching. Very little is known about the multi-dimensional characteristics of assimilated STEM education and efficacious ways of integrating STEM instruction [10]. While research has been done on how to improve students’ problem-solving abilities in STEM education [11] few studies focus on factors that make STEM integration more likely to improve learning outcomes [12]. Hence additional research on instructional design and implementation models is required to provide a real path for improving these skills among students. Using a problem-solving strategy, exposing students to real-world challenges, and achieving the subject’s goals are not possible given the limited class time. Educators, therefore, sought a method of blended learning that would improve face-to-face contact iJIM ‒ Vol. 17, No. 10, 2023 5 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach while allowing students to participate in interactive activities that would develop problem-solving skills. One method of education is the flipped classroom or FC approach. Teacher’s scaffolding with technology integration demonstrated the efficacy of the FC approach by allowing students more time to practise learning in real-world situations [13]. Despite FC having grown in popularity over the past few years, it is still primarily used by individual teachers and not as a standard pedagogical approach. Teachers, on the other hand, have expressed a desire for a credible model depicting the action process in this area [14]. In addition, extensive studies on instructional design and implementation approaches are still required to provide a clear path for students to enhance their problem solving ability using FC [15]. Furthermore, the integrated interdisciplinary STEM approach is still in the embryonic phase and hence development of STEM integration guidelines especially through the FC approach will take precedence in this area to improve learning process practices [16]. Therefore, this current study is specifically to obtain new insights and deliver them as contributions that can be applied by education authorities of K-12 (education from kindergarten to 12th grade) and other stakeholders involved. This study aims at constructing an engineering-based module relying on expert consensus on the module’s aspects and elements, which can be considered as guidance for educators to display STEM curriculum by linking scientific content to real-world issues and suggesting student-centred teaching strategies to improve primary students’ problem-solving skills. The sections that follow will present the theoretical background of the study followed by the research objectives, the research methods, the main findings, and the implications. Lastly, recommendations for future research will be discussed. 2 Theoretical background Some predictions related to integrating engineering design in STEM school subjects mainly talk about the effectiveness of this approach in transmitting the educational message and improving student learning. Implementing integrated STEM in the classroom is a gateway for students to further disseminate ideas and develop critical thinking and problem-solving abilities that will inspire and facilitate their creativity. In other words, in the 21st century, designing frameworks for integrated STEM forms and translating them into practical methods of instruction must be established [17]. Students must understand the interrelationships between STEM disciplines, with the caveat that these relationships are challenging. As a result, integrated STEM curricula must provide concrete examples of how disciplines are integrated besides facilitating application of subject areas in integrated contexts. Constructing a framework for how STEM integration will look like and be implemented in a particular context through the consensus of field experts is essential. This section synthesises the literature to discuss the theoretical basis of the study. Supported by the first principles of instruction theory with underpinnings of the engineering-design model as one of the ways to solve challenges in the fields of STEM, this study benefits from a flipped classroom as a supported environment. Researchers 6 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach seek to form a basic foundation upon which experts rely in including the elements of the instructional module, which aims at improving student problem-solving skills. 2.1 Engineering-based model Engineering design is defined as “an organized, intelligent process in which designers generate, evaluate, and assign concepts for devices and processes whose forms and functions achieve certain objectives or requirements while satisfying specific constraints” [5]. Design is widely regarded as “the essential engineering activity” [18]. The characteristics possessed by the engineering design process as “a method for addressing problems in STEM fields” and the effectiveness of the engineering design serves as a bridge to practical comprehension of STEM content [19]. It has become a requirement to be integrated into the curricula and improve students’ problem-solving abilities [20, 21]. As one of the successful Engineering Design Models (EDM), the Massachusetts department [22] developed a nine-step guide for pre-kindergarten through high school educators and curriculum coordinators (Figure 1). Fig. 1. Modified Engineering Design Process for upper elementary and beyond [22] p. 100 This engineering design model (EDM) has the characteristics of a desirable model, which includes: problem and context comprehension; idea generation; learning novel concepts required to solve problems, creating and applying prototypes, as well as participatory analysis, reviewing and thinking, and revising until a problem is solved. This is known as the engineering design process [23]. Furthermore, during the design process, students must manage uncertainty and risk, analyse previous experiences, and learn from mistakes [24]. Many predictions were made regarding use of the Engineering Design Model in the school setting, mainly showing the revolution in the ways designers and educators can design learning activities to improve student problem-solving skills. A qualitative study [25] was conducted on sixth-grade students participating in bridge modeling activities guided by open-ended geometry-based problems, integrating the activities into STEM curricula. The study revealed that after applying the engineering design model, many students were able to solve problems of varying levels of complexity. iJIM ‒ Vol. 17, No. 10, 2023 7 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach English et al. [26] conducted a three-year longitudinal study highlighting the importance of the engineering design model in enhancing student problem-solving skills and facilitating learning. The study involved 136 sixth-graders who were tasked with completing an integrated STEM-based engineering problem related to earthquakes. The students were required to plan, sketch, and construct earthquake- resistant structures using engineering design processes and STEM subject matter. The activities were designed by a team comprising six experienced teachers, practicing engineers from various fields, and graduate engineering students from a university. The study’s findings revealed that the students demonstrated a clear understanding of STEM concepts and were able to consider multiple components of the problem, indicating their ability to deal with the task complexity. A conceptual framework that has been proposed for secondary education integrated STEM was suggested [21], as an integrated system; the framework includes engineering design, scientific inquiry, technological literacy, and mathematical reasoning as examples of skills. Using the framework, students can test their scientific abilities by applying them to practical problems based on situated cognition theory and community of practice engagement in both engineering and technology. Despite the comprehensiveness of previously proposed frameworks and their consideration of multiple aspects when presenting STEM integration, they are proposals submitted by a small number of experts based on results of previous literature, and they fail to address the details associated with the framework's components and the educational process elements. The majority of previous research was conducted on high school and undergraduate students, whereas the incorporation of a design-based initiative to STEM at the elementary level presents numerous challenges for educators who frequently lack pedagogical content knowledge in STEM. In general, the research on STEM for primary grades appears to be insufficient [27]. 2.2 STEM education STEM education begins in kindergarten and continues to grow through high school. Consensus exists that introducing STEM in the early stages is more successful and acceptable; this is because primary school children demonstrate less anxiety when learning demanding design-based science [24]. It is also vital to introduce children to STEM disciplines or STEM literacy; since middle and high school students begin to consider and choose future careers [20, 21] and use STEM knowledge and skills to solve problems and create products [28], any changes in the decision at this time will have lifelong repercussions. To be able to address critical life problems students need to use expertise across the STEM fields. Integrated STEM education offers real learning situations and makes it possible for students to establish connections between STEM disciplines, besides supporting the development of knowledge and abilities within and across disciplines [29]. From this point appeared the so-called term integrated STEM disciplines and its importance while working on solving authentic problems. Several attempts have emerged to develop conceptual frameworks around the forms of STEM integration. One of them was a model suggested by [30] which defined six 8 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach forms of integration: discipline-based (different subjects taught separately), parallel disciplines (each discipline linked to the same subject or field), multidisciplinary disciplines (some disciplines taught together), interdisciplinary units (deliberately linking disciplines), integrated day (themed disciplines or issues arising from the world of children), and complete programme (completely integrated programme, curriculum designed from student daily life). On their part, [16] proposed a conceptual framework for integrating STEM; the authors propose a framework consisting of four dimensions: 1. Disciplinary Integration: Integrating the STEM content and practices to promote cross-disciplinary learning. 2. Interdisciplinary Integration: Connecting the content and practices of STEM disciplines in conjunction with other subjects, such as the arts or social sciences. 3. Transdisciplinary Integration: Focusing on real-world issues that necessitate knowledge and skills from multiple STEM fields. 4. Integration of Formal and Informal Learning: Creating opportunities for learning that occur both in and out of the classroom. The [16] framework provides a comprehensive approach for designing and implementing STEM education programmes that go beyond simply teaching the content of each subject in isolation. Instead, it emphasises the importance of integrating STEM disciplines to promote deeper understanding of the interconnectedness of these subjects and to prepare students for real-world problems. While [31] indicates that rich STEM integration experiences should include interactive situations that allow students to understand problems from multiple perspectives, engineering design experiences that provide students with the opportunity to learn from failure, and standard-based mathematics and science content through student-centred activities. These proposals for forms of integration have expanded the horizons of employing integrated STEM curricula in different contexts, and the prevailing belief remains that the greater the number of integrated materials, the more student learning will show improvement and gain realism and seriousness. Until a number of educators such as [32] gave a warning that STEM integration must always be meaningful and clearly defined and that in the case of integration more is not always better. Although these studies are based on a review of the STEM education-specific literature to build a conceptual framework around integrating STEM into education, they are not mainly based on original research. As such, the framework is more a synthesis of existing research than a new theory or paradigm. Therefore, it was necessary to build a conceptual framework based on teaching and learning theories and benefit from the opinions of experts to provide valuable insights and perspectives that can inform development of the components of an integrated STEM framework. Moreover, further studies are needed to deepen understanding of how curriculum and instruction can be applied in different contexts and how they can be used to promote meaningful learning [16]. iJIM ‒ Vol. 17, No. 10, 2023 9 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach 2.3 Flipped classrooms Education in Jordan is dense with material. In addition, primary school students lack the time to develop functional reading, writing, and arithmetic skills [33]. In light of this, Jordan must prioritise a solid foundation in essential competencies over the content. Consequently, it was necessary to implement e-learning as a proven alternative to saving classroom time in order to concentrate on essential skills. Undeniably, the Covid-19 pandemic has hastened the shift to e-learning, where comprehensive platforms have been developed to provide online educational content, and the teacher is eager to achieve maximum student interaction. This will not be sufficient, as certain skills (such as problem-solving), however, must be acquired through face-to-face interaction in which the student exercises the skill directly under teacher guidance [34]. Hence the flipped classroom (FC) method in blended learning was regarded as one of the most suitable educational solutions for bridging these gaps. FC method allows digital-age students to be instructed in a nontraditional manner. Therefore, the FC vision was presented as a two-phase model: the first phase consisted of an online tutorial before school, and the second phase consisted of face-to-face meetings and events to promote concept mastery [14]. During the time before class, students view a simple video at home at their own pace. With the aid of their classmates and instructors, students can practice learning in real-world situations and solve real- world problems [13]. Research indicated that in secondary and higher education, the flipped classroom methodology outperforms other methods in terms of learning achievement [35]. Practitioners of this method believe that it improves the teacher-student relationship, allows for deep learning through active classroom participation, and it improves learning outcomes [36]. A number of teachers, however, hesitate to use this method because the design and implementation of a conceptual framework for FC are complicated since it consists of numerous educational components [36]. According to the teachers, the design and development of FC instructional materials and activities, particularly the creation of online learning materials, videos, and related exercises, takes a long time [37]. Furthermore, it is difficult for them to consider all of the contextual factors (e.g., learner characteristics and pedagogical skills) that can have a detrimental effect on FC efficacy [38]. Moreover, substantial differences exist between design contexts. Designers and teachers face obstacles when designing online lectures with appropriate educational and technical specifications, in order to enable students to understand the subject before engaging in classroom activities [39]. Face-to-face design necessitates adequate preparation to guarantee the interactive learning experiences that educators seek. This study is timely as a gap exists in the literature on theoretical, instructional, and methodological approaches to learning, as well as exploration of more situated, observational aspects of the FC approach in schools [40]. 10 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach 2.4 Merrill’s five first principles of instruction Merrill proposed the five first Principles of Instruction theory in 2002, in order to incorporate instructional design principles as a strategy shared by a variety of instructional design models and theories and strongly influenced by constructivism; the first principles of Instruction (FPI) sum up the five prerequisites for establishing an effective learning environment. These principles focused on learning how to overcome real-world challenges. After the problem definition and activation phase, demonstration, application, and integration followed, as summarised in Table 1. Table 1. First Principles of Instruction “as summarised by [7]” Principles Main Significances Problem-centred Students should be engaged in solving problems from the real world. Teachers should assign tasks that students are able to complete. Activation Students should be instructed to recall, relate, describe, or apply their prior knowledge from relevant experiences, which can then serve as the basis for acquiring new knowledge. Demonstration Teachers should demonstrate new knowledge by illustrating concepts, demonstrating procedures, and visualising processes with examples. Students should be given appropriate direction, such as directing them to sources of pertinent information and multiple examples of applied practice. Application Students must apply their newly acquired knowledge or abilities to solve new problems. Teachers should design and sequence problem-solving activities with a variety of challenges. Integration Students should have the opportunity to demonstrate their new knowledge or skills in public. Students must consider, defend, and apply their newly acquired knowledge or skills. Merrill’s first principle of instruction (FPI) is the instructional design model’s most often cited meta theory. It offers educators a potentially beneficial framework for many contexts, especially when implementing a flipped-classroom approach [41]. The FPI appears to be a strong cognitive strategy employed both directly and indirectly through self-direction, motivation, and performance in course-level implementation [42]. By using FPI learners exhibited a greater increase in learning at the level of recall and were more convinced in solving future problems. A researcher [43] conducted two exploratory studies on FPI application as a foundation for flipped classroom frameworks; In Study 1, low-performing students were offered flipped STEM classrooms as a treatment technique, while in Study 2, the impacts of a flipped classroom strategy were investigated for exceptional ability students. Results indicated, that this framework can help boost STEM performance for both underperforming and gifted students. Very few studies, however, have been undertaken on instructional designers’ application of the First Principles of Instruction in their design decisions. Many FC studies lack robust theoretical foundations for their instructional designs [44]. especially when choosing activities [20]. Thus, without a theory-driven framework, FC efficacy may be affected unexpectedly. Hence, for each educational activity, the first instruction principles in a FC need greater clarification. Studies on developing iJIM ‒ Vol. 17, No. 10, 2023 11 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach programmes for integrating STEM usually fail to explain various educational principles behind their design (for example, see [45]). Thus, the current FSC engineering-based module used Merrill’s First Principles of Instruction to help professionals choose FC activities to enhance student problem-solving skills. 3 Statement of the problem The national innovation system found that research in Jordan was contributing insufficiently to economic growth and the resolution of real-world challenges, particularly those involving STEM [46]. Students fail to adopt an acceptable problem- solving method thus contributing to their ineffectiveness in dealing with real-world issues [3]. Since education in Jordan is content-intensive, students in elementary school lack time to build functional competencies, since they are overloaded with irrelevant knowledge. This means that when more rigorous learning is required, such as STEM, students lack the essential learning competencies, and STEM education is consequently hampered from the start. Jordanian teachers lament a lack of class time to teach generic skills such as problem-solving. Hence the flipped classroom is designed to integrate technology while the student follows the teaching in an asynchronous mode, allowing for more active learning and teaching through complex problem-solving activities. To assist students, educators are required to provide examples of solid instructional procedures, especially when teaching knowledge, intellectual, and complex disciplines such as science and mathematics. One method for achieving this objective is to contextualise STM by selecting engineering experiences in authentic circumstances. By inventing and constructing procedures, students can achieve “conceptual cohesiveness” to facilitate learning [4]. In Jordan, however, technology integration in teaching is still in its early stages, and additional research is essential to support more extensive use of the flipped classroom. On the other hand, more research on instructional design and implementation models is required to give a realistic path for improving skills among students [15]. Therefore, Merrill’s fundamental principles of teaching have been utilised in this study as a guide for choosing acceptable learning activities to improve primary student problem-solving skills. 4 Purpose of the study This research intends to design a flipped STEM classroom engineering-based module for Jordanian seventh graders. The objective of this study is to validate the module aspects and elements based on expert consensus. This study is expected to help grade seven students enhance their problem-solving skills in integrating knowledge in resolving complex problems, particularly in STEM. Therefore, the research question guiding the study was: What are the panelists’ opinions on the design and development of the FSC engineering-based module in enhancing 7th-grade students’ problem-solving skills? 12 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach 5 Methodology This is the second phase of design and development research undertaken in the spring of 2021, based on the findings of the previous phase before designing this module, which is the learner analysis, as explored in a study [2]. In the first phase, the Problem-Solving Inventory (PSI) was used to obtain information regarding student perceptions of their problem-solving abilities from 120 Jordanian female students in the seventh grade of a private school. The analysis results of the first phase indicated that the students had moderate general perceptions of their capacity to tackle real-world challenges The findings showed the need for implementing a multi-level intervention to increase students’ problem-solving awareness. In order to continue on the initial phase of analysing student needs, this study extends to the construction of a module that promotes student problem-solving abilities. In this investigation, the mixed mode approach of data collection and analysis was followed utilising the Fuzzy Delphi Method. FDM was implemented in two phases: First, an interview was carried out to obtain experts’ views on the module aspects and elements. Second, the Fuzzy Delphi questionnaire was employed to get the agreement of other experts on the chosen most favoured module aspects and elements. The module was then developed based on FDM results from the panel of experts of the two phases. 5.1 Participants The Fuzzy Delphi Method will emphasise construction of an FSC engineering-based module through the opinions of a panel of module specialists. In the beginning, a semi- structured interview was carried out to obtain views from nine experts. This interview generated views of the panel of experts on the module aspects and their elements. Second, the FDM was employed to get agreement of 29 experts on the chosen most favoured aspects and their elements of the flipped classroom module to solve STEM problems through the engineering model. The module was then developed based on FDM results from the panel of experts. All experts were selected using purposive sampling. They were chosen in light of their backgrounds and expertise in several areas, as shown in Table 2. Table 2. Expertise of experts for FDM Experts NO. Position Expertise STEM Curriculum and Instructional design Technology in education Engineering and building Design Problem Solving 3 Professor- public university / / / 8 Lecturers-public universities / / / 14 Senior lecturers public universities / / 8 Teaching assistant of engineering - public university / / / 5 Professor – private university / / iJIM ‒ Vol. 17, No. 10, 2023 13 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach 5.2 Instrument validity and reliability Fuzzy Delphi instruments consist of 2 parts; the interviews with the first panel of experts and the FDM questionnaire. The interview is transcribed and the data sent to the respondents for validation to ensure data accuracy. A member check is performed to compare the data with the original responses. Then each interview was analysed through a process of coding and condensing the codes to reduce the collected data into themes. The interviews were transcribed and the emerged themes were employed for designing the FDM items. After nine experts responded to the interview questions, the FDM questionnaire constructed was then presented to another 3 experts to ensure validity. The Fuzzy Delphi Method was employed to get the agreement of 29 experts on the chosen most favoured module aspects. The module was then developed based on FDM results. A questionnaire with a linguistic seven-point scale evaluating the preferences of experts consists of two sections: Expert Demographic List and Module Aspects, which includes the form of STEM integration, assessment, resources, and pre-class, and in-class instructional activities. The Cronbach alpha value was used to assess the questionnaire reliability, and it was found to be reliable (.75). 5.3 Data analysis The FDM questionnaire was distributed to a panel of 29 specialists, then data analysis was followed according to the procedures of Jamil et al. in their study [47]. Step 1: Using a triangular fuzzy number, determine the linguistic scale. As can be seen in Table 3, the linguistic variables of the questionnaire have been converted into a fuzzy triangular number. In order to take into consideration, the uncertainty of the experts’ points of view, each response was given one of three fuzzy value categories. The greatest value (m3), the most reasonable value (m2), and the smallest value (m1) are depicted in Figure 2 as the different levels of fuzzy value. m1 = “minimum value”; m2 = “most plausible value”; m3 = “maximum value” Fig. 2. Shows Triangular Fuzzy Number 14 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach The linguistic variable can be converted into fuzzy numbers by using the linguistic scale. It is imperative that the agreement’s level scale contain an odd number of tiers. The response analysis will yield more precise data when the scale is increased. A linguistic scale with seven points is displayed in the fuzzy scale in Table 3. Table 3. Seven-Point Scale Linguistic Variables Linguistic Variable Fuzzy Scale Very strongly agree 0.90 1.00 1.00 Strongly agree 0.70 0.90 1.00 Agree 0.50 0.70 0.90 Moderate agree/ Not sure 0.30 0.50 0.70 Disagree 0.10 0.30 0.50 Strongly disagree 0.00 0.10 0.30 Very strongly disagree 0.00 0.00 0.10 Step 2: Determine the mean value (a1, am, a2). To determine the median replies for every fuzzy number, the following formula was utilised to compute them: (1) This is done in order to determine the value that is typical of m1, m2, and m3. The average is then used in computing the threshold value, which results in the a1, am, and a2 values being obtained. Step 3: Establish the threshold value (d). The threshold value, denoted by the letter d, was arrived at by calculating the disparity between the rating data supplied by the experts and the average value for each element. (2) The threshold value is one of two module acceptance criteria. “To consider expert group consensus, the resulting threshold (d) must be smaller than or equal to 0.2 (d ≤ 0.2)”. This study uses three decimal points, thus any item with a threshold value (d) below 0.3 will be considered an expert agreement. Condition 2 involves an expert percentage. The Delphi method’s traditional approach affects the percentage by the number of items with a threshold value (d) of 0.3 or higher. Thus, Delphi is used to convert items having a (d) threshold value of 0.2 or less into a percentage. Step 4: Defuzzification. Defuzzification is performed to validate the scores or rankings of the elements. Its objective is to achieve a fuzzy score (A), to achieve a consensus among all experts. Using the following equation, the defuzzification value (DV) is calculated for each survey item (module element) by: Amax = 1/3 * (a1 + am + a2) iJIM ‒ Vol. 17, No. 10, 2023 15 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach The median value (α - cut) of 0.5 must be met or exceeded by the fuzzy scores (A) that should be greater than or equal to that number. The items that were measured are discarded if the fuzzy (A) score value is lower than 0.5, as determined by the general agreement of the expert group. 6 Findings The results of the FDM questionnaire were analysed using the Excel program. Each element’s defuzzification value was calculated, and the consensus was determined using a threshold value of (d) = 0.75. A fuzzy score of > 0.5 was used to select the online and in-class phases of the module that needed highlighting for implementing the instruction. Table 4. Panel experts’ consensus on the most preferred elements of the FSC engineering- based module Panel Experts’ Consensus on the Most Preferred form of STEM integration No. A B C D 1 Interdisciplinary: organise the curriculum around common learnings across disciplines. Chunk together the common learnings embedded in the disciplines to emphasise interdisciplinary skills and concepts. 0.094 86.2% 0.875 2 Multidisciplinary: focusing primarily on the disciplines. Organise standards from the disciplines around a theme 0.108 75.9% 0.911 3 Transdisciplinary: organise curriculum around student questions and concerns 0.159 82.8% 0.852 Panel Experts’ Consensus on the Most Preferred Assessment No. A B C D 1 Performance-based assessment 0.032 86.2% 0.956 2 Quiz/Test 0.300 41.4% 0.370 3 Online discussion 0.160 82.8% 0.794 Panel Experts’ Consensus on the Most Preferred Resources No. A B C D 1 Books on engineering-based processes in STEM disciplines 0.270 55.2% 0.723 2 Articles on engineering-based processes in STEM 0.155 79.3% 0.856 3 Web resources to find out more information about engineering-based processes in STEM 0.114 86.2% 0.876 4 YouTube videos on links to Web resources to find out more information about engineering-based processes and methodology 0.161 79.31% 0.775 Panel Experts’ Consensus on the Most Preferred online platform No. A B C D 1 Website 0.369 27.6% 0.547 2 Blog 0.249 65.5% 0.660 3 Facebook 0.315 41.4% 0.409 4 Learning Management System (LMS) 0.131 82.76% 0.877 16 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach Panel Experts’ Consensus on the Most Preferred Media Elements No. A B C D 1 Text 0.193 82.8% 0.817 2 Animation 0.092 82.8% 0.930 3 Graphics and Images 0.070 82.8% 0.940 4 Music 0.289 44.83% 0.617 5 Audio narration 0.158 79.31% 0.756 Panel Experts’ Consensus on the Most Preferred Duration of the pre-class video No. A B C D 1 Last to six minutes 0.032 86.2% 0.956 2 From 6 to 10 min 0.344 27.6% 0.239 3 lasting more than 10 min 0.098 79.3% 0.059 Panel Experts’ Consensus on the Most Preferred video presentation No. A B C D 1 View a complete and consecutive video, and ask questions at the end of the show 0.139 82.8% 0.895 2 View a complete and consecutive video that includes clips, and ask questions at the end of each clip 0.370 41.4% 0.446 Panel Experts’ Consensus on the Most Preferred Method of answering the online exercises No. A B C D 1 The answers to the exercises are sent to the teacher’s account and the only one who gives feedback 0.354 41.4% 0.251 2 The answers to exercises are written below the video, discussed and debated by other students and teachers 0.070 82.8% 0.928 3 The answers to exercises are written below the video, teacher is the only one who gives feedback 0.309 41.4% 0.209 Panel Experts’ Consensus on the Most Preferred online exercises No. A B C D 1 Simple questions need answers to make sure students understand the video content 0.214 79.3% 0.831 2 Simple questions that gradually become a higher level to solve well-structured problems that need a direct solution. 0.086 82.8% 0.913 3 Surveys are used to ask students about their concerns regarding lesson concepts. 0.209 79.3% 0.794 4 Online projects are to be done and shown as websites/ blogs for the class and community to view. 0.075 82.76% 0.924 Panel Experts’ Consensus on the Most Preferred instructional events of in-class short didactic lectures No. A B C D 1 (Activation) Start with a brief review of the out-of-class learning from the video 0.049 86.2% 0.939 2 (Demonstration) Review students' online responses. Demonstrate new knowledge. Guiding students by directing them to sources of relevant information and to multiple demonstrations of applied practice 0.075 82.8% 0.923 Panel Experts’ Consensus on the Most Preferred instructional events of in-class interactive activities iJIM ‒ Vol. 17, No. 10, 2023 17 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach No. A B C D 1 (Problem-centred) Engage students in solving real-world problems; provide tasks that the students will be able to do 0.079 82.8% 0.914 2 (Application)- Identifying the problem, improve and/or fix through recalling, relating, or applying students’ previous knowledge -Using new knowledge or skills to solve new problems, by modeling possible solutions, and refining models. Testing the proposed solution 0.066 86.2% 0.929 3 (Integration) Communicating, explaining, and sharing the solution and design. This allows the students to apply their new knowledge or skills. 0.073 86.2% 0.925 A: Elements B: Average response Fuzzy evaluation C: Alpha-Cut Fuzzy score D: Threshold value The FDM wanted to best determine the expert panel’s preferences for FSC engineering-based module elements. The defuzzification value obtained for each item must be ≥ (0.750), as the item with the highest agreement percentage is selected. As a result, the elements shown in Table 5 have been agreed upon. Table 5. Elements of the engineering-based flipped unit in the STEM classroom as agreed upon by the expert group Aspect Elements Average response Fuzzy evaluation Alpha-Cut Fuzzy score (A) Threshold value (d) Form of STEM integration Multidisciplinary: focusing primarily on the disciplines. Organise standards from the disciplines around a theme 75.9% 0.911 0.108 Assessment Performance-based assessment 86.2% 0.956 0.032 Online discussion 82.8% 0.794 0.160 Resources Articles on engineering-based processes in STEM 79.3% 0.856 0.155 Web resources for learning more about engineering-based processes in STEM 86.2% 0.876 0.114 YouTube videos on links to Web resources for learning more about engineering-based processes and methodology 79.31% 0.775 0.161 Pre-class instructional events: Online platform Learning Management System (LMS) 82.76% 0.877 0.131 Media Elements Text 82.8% 0.817 0.193 Animation 82.8% 0.930 0.092 Graphics and Images 82.8% 0.940 0.070 Audio narration 79.31% 0.756 0.158 Duration of the pre-class video Last up to six minutes 86.2% 0.956 0.032 Video presentation View a complete and consecutive video, and ask questions at the end of the show 82.8% 0.895 0.139 Method of answering the online exercises The answers to exercises are written below the video, discussed and debated by other students and teachers 82.8% 0.928 0.070 18 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach Aspect Elements Average response Fuzzy evaluation Alpha-Cut Fuzzy score (A) Threshold value (d) The type of online exercises Simple questions gradually become a higher level to solve well-structured problems that need a direct solution. 82.8% 0.913 0.086 Online projects to be done and shown as websites/blogs for class and community to view. 82.76% 0.924 0.075 The in-class instructional events: Instructional events of short didactic lectures (Activation) Begin by reviewing the videos for out-of-class learning. 86.2% 0.939 0.049 (Demonstration) Examine the online responses of the students. Demonstrate fresh understanding. 82.8% 0.923 0.075 The interactive activities Students are guided by pointing them to sources of pertinent information and many examples of actual practice (Problem-centred) Students were engaged in tackling real-world challenges. Provide assignments that pupils will be able to complete. 82.8% 0.914 0.079 (Application)- Identifying the problem, improving and/or fixing it through recalling, relating, or applying students' previous knowledge. Using new information or abilities to solve new difficulties. by modeling possible solutions, and refining models. Testing the proposed solution 86.2% 0.929 0.066 (Integration) Presenting, communicating, and discussing the solution and design. This enables students to use their newly acquired knowledge or abilities. 86.2% 0.925 0.073 According to Table 5, the expert panel selected by consensus the final module elements agreed upon from a set of elements previously proposed by the interviewed experts. Only the elements possessing a certain value (defuzzification value of 0.877) and the threshold value (d) ≤ 0.2, and the fuzzy score (A) ≥ α - cut value = 0.5 were approved, while the other suggestions were rejected based on experts’ preferences. The findings led to the selection of 21 elements from the five aspects of the FSC engineering-based module. Based on FDM findings and following the literature, the researchers created the module framework to include Merrill’s first principles of instruction [7] and the engineering design model [22]. As illustrated in Figure 3, this demonstrates the connection between the instructional events before and during classes, where the Massachusetts model is integrated into the implementation and integration phases of Merrill’s first principles of instruction to solve STEM-related actual problems during face-to-face sessions. Massachusetts engineering design model [22]. As illustrated in Figure 3, this demonstrates the connection between the instructional events before and during classes, where the Massachusetts model is integrated into the implementation iJIM ‒ Vol. 17, No. 10, 2023 19 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach and integration phases of Merrill’s first principles of instruction to solve STEM-related actual problems during face-to-face sessions. Fig. 3. A description of the flipped classroom method that combines Merrill's first principles of instruction [7] with the engineering design model [22] The instructional framework includes two major components: (1) pre-class computer-based learning and (2) in-class active learning, with the aid of Merrill’s First Principles of Instruction [7] and engineering-based model of Massachusetts [22]. The following is a detailed presentation of the general framework and the contents of the module. 6.1 Pre-class computer-based learning Pre-class learning activities sufficiently equip learners for classroom learning, without taxing them. The four phases of Merrill’s First Principles are the focus of pre- class lesson planning: activation, demonstration, application, and integration. Two essential instructional activities (pre-class video lesson and online exercises) make up the pre-class learning component. Pre-class video lesson: Students are exposed to the learning content through videos during pre-class learning. Students given video lessons came to class much better prepared than those given textbook readings [48]. Teachers start by showing a few review videos to revisit the prerequisite background knowledge for learning new concepts (i.e., activation principle). The students could then review previous knowledge based on their individual requirements. Following that, the teachers introduce new information or show examples (i.e., demonstration principle). Independent student learning should be sufficiently scaffolded so that students can share their inquiries and find areas of difficulty for teachers to clarify during the lesson. 20 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach Online exercise: The online questions were provided to ensure students understand the video content and apply their new understanding and pass on their knowledge. By using teacher-created computerised feedback the students can evaluate their learning progress. If students incorrectly answer the online questions, hints or feedback can be offered to guide them to the right resources (i.e., demonstration principle). The video lectures can then be reviewed by students for a more in-depth understanding. The class assignment also was conducted to ensure preparation for the in-class lesson. 6.2 In-class interactive learning Classroom learning includes two main parts; the small didactic lectures and the problem-solving activities. Small didactic lectures: At the beginning of class meetings, teachers start lessons with a small didactic lecture by introducing a brief summary of the outside-of-class learning (activation) to identify misconceptions according to the video contents. Because not all course materials are appropriate for self-directed learning through instructional videos, it is also essential to present some of the more complicated concepts (demonstration). Problem-solving activities: Teachers start activities with fundamental questions on problems to be identified (identify needs) through the real-world context (i.e., application). This problem relates to engineering design which clarifies the links between learning and the real world to develop problem-solving and self-directed investigation skills. Problem definition involves moving from a broad statement of need to a specific problem, determining criteria for a successful solution, and identifying limitations, the math, and science required to analyse data and formulate the solution, through knowledge of the underlying science and mathematising the qualitative measures of success. Students collect the necessary information to understand the problem, and this leads to understanding the math and science concepts central to the challenge while tackling their engineering design. The students brainstorm multiple possible solutions and develop systems for choosing between those solutions. After proposing multiple solutions, students engaged in building models/prototypes into which the solution must fit. After that, students have to test the solutions to decide whether the existing solutions/prototypes come to grips with the underlying constructions. Students work to transfer the idea of the project and the solution method to their colleagues and even to the external community (solution integration) to transfer experiences and take feedback on it. This process requires that the solution be reviewed to be supported and better adapted to fit the criteria and restrictions, or that it can be developed to solve other realistic problems and this gives an iterative feature (redesign and revise) of the engineering model. iJIM ‒ Vol. 17, No. 10, 2023 21 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach 7 Discussion, and recommendations In this study, five primary components, and their 21 constituent elements were agreed upon by a group of experts to develop a module based on solving STEM problems for primary school students. The aspects and elements of the proposed module will be discussed here. Regarding the form of STEM integration, multidisciplinary approaches were recommended. Now overall real-world issues are too vast to be comprehended by a single field, necessitating a multidisciplinary [49] approach, as it is the most feasible and offers various academic disciplines in a thematically-based study with multiple objectives. Experts agree that the assessment method for this module is a performance-based assessment with online discussion. Students will be evaluated collectively in STEM subjects to demonstrate that they understand scientific concepts and can apply their knowledge to examine the natural world through scientific inquiry or to solve real- world problems using engineering design methodologies [23]. The performance-based assessment is the recommended method for evaluating students’ STEM knowledge using a variety of tasks and methodologies. According to [50], experts concur that student involvement in the educational platform is crucial because it ensures that students comprehend the lesson’s core elements. The FSC engineering-based module’s shared resources include articles and Web resources, such as YouTube, for gaining additional information on engineering-based STEM procedures. Otherwise, the option regarding the books was rejected due to the paucity of engineering design books and the limited availability of free books for both instructors and students. Learning Management System (LMS) platforms, especially Open-source platforms, are the most common online platforms for video lectures in instructional events of a pre-class lesson to demonstrate the module online components. They assist consumers by permitting platform change based on user requirements and by charging reasonable rates for enhanced service [51]. The considered Facebook option was rejected because it was perceived as posing a threat to productivity [52]; users, particularly elementary school pupils who are less responsible for their own learning, might waste time conversing, playing games, and viewing images. Experts have consistent opinions about using websites, blogs, and Facebook as module resources because they do not offer the same capabilities as learning management systems (LMS) for test and assignment preparation, storing videos and documents in the library, and ensuring student privacy during access to pages displaying their assessments. As for the media elements, the experts decided on the integrated media features and the video sequence, which consists of graphics and images, animation, text, and voice- over. This is corroborated by a survey-based study [53] which revealed that multimedia use in animation movies considerably improved the quality of learning materials for engineering drawings. Engaging students in the lecture also helps to direct their attention, allowing them to process certain video components in working memory [54]. More than half of the experts are in favour of introducing music into educational videos, while 10% are unsure and 23.3% oppose it. According to experts, poor theme selection can distract students from their studies. Music is sometimes viewed as 22 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach interesting, but it adds little to learning objectives; learners must decide whether to pay attention to it, which increases the external memory burden and can occasionally obstruct learning [54] and impacts memory in general [55]. In fact, [54] suggested that background music was detrimental to memory recall in multimedia learning. Therefore, the music elements did not meet expert agreement. As for the online video duration, experts agreed that it should be roughly six minutes. Guzey et al. [56] examined data from 6.9 million video sessions to determine how much time students spent watching streaming videos in four edX MOOCs. They found that the average video viewing time was less than six minutes and that nearly all students viewed the entire film, indicating that students were inclined to do so. While [57] used auto videos as part of their 3D modeling technique in an eighth-grade ICT course and reported that instructional videos longer than 10 minutes led to student disengagement. It was agreed that the video presentation would be shown, and students were given the opportunity to ask questions thereafter. This guarantees that the video content is broadcast prior to questioning the pupils; [55] confirmed that the majority of students stated that viewing recordings of practical sessions and responding to brief questions was advantageous. Moreover, the instructional film and accompanying exam questions motivate students to achieve higher learning outcomes and positively improve their learning experiences. The experts, however, agreed that the preferred technique for completing the online assignments is for students to write their responses beneath the video and then discuss and debate them with other students and instructors. As students engage in discourse with one another and the instructor, this has a good impact on learning and improves learning outcomes. Only two types of online tasks in the pre-class session have been selected according to agreement of experts as having the highest fuzzy score value. This is important so that pupils do not become confused by the several techniques. After completing the engineering challenges, students will complete online projects and post them on public websites. Another sort of online activity involves students viewing and posing easy questions that progressively get more difficult in order to address directly well-structured problems; [58] stated that online activities enabled students to use their newly gained knowledge, hence encouraging them to keep learning. The panel of experts’ consensus-related guiding principles offered by Merrill [7], that the in-class lesson consists of two sections; Short instructional lectures and classroom problem-solving activities. In-class brief instructional lectures are delivered at the start of class meetings. [15] emphasised that learning improves if the teachers started their lessons by reviewing outside learning (i.e., activation principle) briefly. This will help students later with problem-solving and group discussion. A question- and-answer session could enable teachers to dispel any misunderstandings pupils may have regarding video content (i.e., demonstration). In addition, the module introduces the application principle, which allows teachers to enrich and encourage exercises that allow students to use their knowledge. Using an engineering design model to provide students with a comprehensive approach to problem-solving is a natural application across all STEM fields [21]. Consequently, students had more opportunities to apply their knowledge to problem-solving, particularly in flipped sessions (for example, see iJIM ‒ Vol. 17, No. 10, 2023 23 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach [59]). During class meetings, teachers may provide formative assessment and additional scaffolding while students work with peers. Lastly, the integration will enable students to convey, explain, and share the solution and design of their newly acquired information or abilities. [15] stated that, by describing a topic or concept to their peers, students could learn it more thoroughly. The method of learning-by-teaching, which allows the student to assume the role of the instructor, and reflect on the learning process itself are characteristics of flipped classrooms [60]. In order to design an appropriate, relevant, and useful FSC engineering-based module to improve problem-solving abilities and other crucial efficient learning practices, it is crucial to confer with a panel of experts to collect contextual information data utilising FDM. 8 Conclusion This paper discussed the findings of a study aimed at identifying aspects and elements of the FSC Engineering-based module based on the experts’ consensus. The results of the study show that a general consensus of experts was obtained to determine the module structure, that is, the instructional design of STEM lessons using the flipped classroom approach, in order to improve problem-solving skills among primary students. Next, this study makes many contributions, especially in the educational technology field. First, the emergence of a new framework that combines two instructional designs, the engineering design model [22] and Merrill’s first principles of instruction [7], thus filling a knowledge gap in this field. The combination of the two theories led to establishing the basics of the module, which is an educational design consisting of the stages of teaching and learning within the flipped classes (pre- classroom and in-classroom) and what they contain of educational events that include content presentation, activities, and assessment. The Fuzzy Delphi method was then applied to confirm the module aspects and elements. A group of experts is involved in this process to reach a consensus on decision-making. The study’s findings can be used as a guideline for educators and instructional designers in developing integrated STEM modules that adapt to student life and help students to solve authentic problems. The inclusion of the engineering design paradigm in STEM education will help create an adaptable community in a fast-changing world. The study emphasises the importance of using the flipped classroom approach to develop educational resources based on STEM integration and validation by experts. To determine the best approach to STEM integration, a research agenda must be developed to test theories under a variety of conditions. In the United States, the Commission on Integrated STEM Education has made several recommendations aimed at various stakeholders in integrated STEM education, including those who design integrated STEM initiatives, those who develop assessments, and researcher educators [8]. For further research into integrated STEM education, researchers should document their interventions, curricula, and programmes in more detail, particularly how they integrate and support subjects. More research is needed to determine the nature of integration, scaffolding, and instructional designs. Clear outcomes regarding how 24 http://www.i-jim.org Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach integrated STEM education promotes learning, thinking, interest, and other characteristics must be identified and measured. The proposed module, in particular, however, must be supported by strong empirical evidence. Hence, the central problem identified in this paper proposes the following basic research questions for future studies: How can this module help school teachers develop integrated STEM curricula? Was the module successful in meeting its own goal of improving seventh-grade students’ problem-solving abilities? Given the limitations of research conducted in one geographical area, it is uncertain whether the findings can be generalised to other settings. Thus, further research in more geographical areas is needed to help researchers understand whether similar trends are evident elsewhere. 9 Declaration “All authors certify that this article is original, has not been previously published, and is not being considered for publication elsewhere at this time.” 10 References [1] G. P. Wiggins, G. Wiggins and J.McTighe, Understanding by design. ASCD. 2005. [2] K. M. M. Naser, S. H. Halili, and R. A. Razak, “Needs analysis in solving STEM problems among Jordanian students,” Journal of Educational Research, vol. 39, pp. 118–126, 2021. [3] A. İ. C. Gözüm, S. Papadakis, and M. Kalogiannakis, “Preschool teachers’ STEM pedagogical content knowledge: A comparative study of teachers in Greece and Turkey,” Front. Psychol., vol. 13, p. 996338, 2022. https://doi.org/10.3389/fpsyg.2022.996338 [4] M. J. Nathan, R. Srisurichan, C. Walkington, M. Wolfgram, C. Williams, and M. W. Alibali, “Building cohesion across representations: A mechanism for STEM integration: Cohesion across representations,” J. Eng. Educ., vol. 102, no. 1, pp. 77–116, 2013. https://doi.org/ 10.1002/jee.20000 [5] S. D. Sheppard, K. Macatangay, A. Colby, L. S. Shulman, W. M. Sullivan, and S. Sheppard, Educating Engineers. Jossey-Bass, 2008. [6] S. E. August, M. L. Hammers, D. B. Murphy, A. Neyer, P. Gueye, and R. Q. Thames, “Virtual Engineering Sciences Learning Lab: Giving STEM Education a Second Life,” IEEE Transactions on Learning Technologies, vol. 9, no. 1, pp. 18–30, Jan. 2016. https://doi.org/10.1109/tlt.2015.2419253 [7] M. D. Merrill, “First principles of instruction,” Educational Technology Research and Development, vol. 50, no. 3, pp. 43–59, Sep. 2002. https://doi.org/10.1007/bf02505024 [8] M. Nathan and G. Pearson, “Integration in K–12 STEM education: Status, prospects, and an agenda for research,” in 2014 ASEE Annual Conference & Exposition Proceedings, 2020. [9] R. W. Bybee, The case for STEM education: Challenges and opportunities. NSTA Press, 2013. [10] N. A. G. Board, Technology and Engineering Literacy for the 2014 National Assessment of Educational Progress. 2010. [11] D. J. Shernoff, S. Sinha, D. M. Bressler, and L. Ginsburg, “Assessing teacher education and professional development needs for the implementation of integrated approaches to STEM iJIM ‒ Vol. 17, No. 10, 2023 25 https://doi.org/10.3389/fpsyg.2022.996338 https://doi.org/10.1002/jee.20000 https://doi.org/10.1002/jee.20000 https://doi.org/10.1109/tlt.2015.2419253 https://doi.org/10.1007/bf02505024 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach education,” Int. J. STEM Educ., vol. 4, no. 1, p. 13, 2017. https://doi.org/10.1186/s40594- 017-0068-1 [12] L. D. English and D. T. King, “STEM learning through engineering design: fourth-grade students’ investigations in aerospace,” Int. J. STEM Educ., vol. 2, no. 1, 2015. https://doi. org/10.1186/s40594-015-0027-7 [13] K. Clark, “The Effects of the Flipped Model of Instruction on Student Engagement and Performance in the Secondary Mathematics Classroom,” The Journal of Educators Online, vol. 12, no. 1, Jan. 2015. https://doi.org/10.9743/jeo.2015.1.5 [14] K. Srivastava, “Role of Flipped Classroom in Education,” Paripex - Indian Journal of Research, vol. 3, no. 4, pp. 81–83, Jan. 2012. https://doi.org/10.15373/22501991/apr2014/ 27 [15] C. K. Lo, C. W. Lie, and K. F. Hew, “Applying ‘First Principles of Instruction’ as a design theory of the flipped classroom: Findings from a collective study of four secondary school subjects,” Comput. Educ., vol. 118, pp. 150–165, 2018. https://doi.org/10.1604/9780787– 977436 [16] Honey, M., Pearson, G. & Schweingruber, H. (Eds.), STEM Integration in K-12 Education: Status, prospects and an agenda for research. Washington DC: National Academies Press, 2014. [17] G. H. Roehrig, E. A. Dare, E. Ring-Whalen, and J. R. Wieselmann, “Understanding coherence and integration in integrated STEM curriculum,” International Journal of STEM Education, vol. 8, no. 1, pp. 1–21, 2021. https://doi.org/10.1186/s40594-020-00259-8 [18] Y. Li et al., “Design and design thinking in STEM education,” J. STEM Educ. Res., vol. 2, no. 2, pp. 93–104, 2019. https://doi.org/10.1007/s41979-019-00020-z [19] T. J. Moore, E. A. Siverling, and K. A. Douglas, “Evidence‐based reasoning: Eliciting STEM content during engineering design justifications,” School Science and Mathematics, vol. 119, no. 8, pp. 1–3, Dec. 2019, https://doi.org/10.1111/ssm.12378. [20] Chen Yong-yi, “A literature review of advancing early STEM education in the GBA,” J. Lit. Art Stud., vol. 12, no. 6, 2022. https://doi.org/10.17265/2159-5836/2022.06.016 [21] T. R. Kelley and J. G. Knowles, “A conceptual framework for integrated STEM Education,” International Journal of STEM Education, vol. 3, no. 1, pp. 1–11, 2016. https://doi.org/ 10.1186/s40594-016-0046-z [22] “Massachusetts. Department of Elementary and Secondary Education,” 2016 Massachusetts science and technology/engineering curriculum framework, Massachusetts Department of Elementary and Secondary Education, 2016. [23] A. Bagiati and D. Evangelou, “Practicing engineering while building with blocks: identifying engineering thinking,” European Early Childhood Education Research Journal, vol. 24, no. 1, pp. 67–85, Jan. 2016, https://doi.org/10.1080/1350293x.2015.1120521. [24] J. G. Wells, “Efficacy of the Technological Engineering Design Approach: Imposed Cognitive Demands Within Design-Based Biotechnology Instruction,” Journal of Technology Education, vol. 27, no. 2, May 2016. https://doi.org/10.21061/jte [25] L. D. English and N. G. Mousoulides, “Bridging STEM in a real-world problem,” Mathematics Teaching in the Middle School, vol. 20, no. 9, pp. 532–539, 2015. https://doi.org/10.5951/mathteacmiddscho.20.9.0532 [26] L. D. English, D. King, and J. Smeed, “Advancing Integrated STEM Learning Through Engineering Design: Sixth-grade students’ design and construction of earthquake resistant buildings,” The Journal of Educational Research, vol. 110, no. 3, pp. 255–271, 2016. https://doi.org/10.1080/00220671.2016.1264053 26 http://www.i-jim.org https://doi.org/10.1186/s40594-017-0068-1 https://doi.org/10.1186/s40594-017-0068-1 https://doi.org/10.1186/s40594-015-0027-7 https://doi.org/10.1186/s40594-015-0027-7 https://doi.org/10.9743/jeo.2015.1.5 https://doi.org/10.15373/22501991/apr2014/27 https://doi.org/10.15373/22501991/apr2014/27 https://doi.org/10.1604/9780787%E2%80%93977436 https://doi.org/10.1604/9780787%E2%80%93977436 https://doi.org/10.1186/s40594-020-00259-8 https://doi.org/10.1007/s41979-019-00020-z https://doi.org/10.1111/ssm.12378 https://doi.org/10.17265/2159-5836/2022.06.016 https://doi.org/10.1186/s40594-016-0046-z https://doi.org/10.1186/s40594-016-0046-z https://doi.org/10.1080/1350293x.2015.1120521 https://doi.org/10.21061/jte https://doi.org/10.5951/mathteacmiddscho.20.9.0532 https://doi.org/10.1080/00220671.2016.1264053 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach [27] D. King and L. D. English, “Engineering design in the primary school: applying STEM concepts to build an optical instrument,” International Journal of Science Education, vol. 38, no. 18, pp. 2762–2794, Dec. 2016. https://doi.org/10.1080/09500693.2016.1262567 [28] G. Falloon, M. Hatzigianni, M. Bower, A. Forbes, and M. Stevenson, “Understanding K-12 STEM education: A framework for developing STEM literacy,” J. Sci. Educ. Technol., vol. 29, no. 3, pp. 369–385, 2020. https://doi.org/10.1007/s10956-020-09823-x [29] National Research Council, Developing assessments for the next generation science standards. 2014. [30] H. H. Jacobs, Interdisciplinary curriculum design and implementation. Alexandria, VA 22314: Association for Supervision and Curriculum Development, 1989. [31] T. J. Moore, M. S. Stohlmann, H.-H. Wang, K. M. Tank, A. W. Glancy, and G. H. Roehrig, “Implementation and integration of engineering in K–12 STEM Education,” Engineering in Pre-College Settings, pp. 35–60, 2014. https://doi.org/10.2307/j.ctt6wq7bh.7 [32] S. S. Guzey, T. J. Moore, and M. Harwell, “Building up STEM: An analysis of teacher- developed engineering design-based STEM integration curricular materials,” Journal of Pre-College Engineering Education Research (J-PEER), vol. 6, no. 1, p. 2, 2016. https://doi.org/10.7771/2157-9288.1129 [33] L. Carter, “Neoliberalism and STEM education: Some Australian policy discourse,” Can. J. Sci. Math. Technol. Educ., vol. 17, no. 4, pp. 247–257, 2017. https://doi.org/10.1080/14926– 156.2017.1380868 [34] K. Lavidas, Z. Apostolou, and S. Papadakis, “Challenges and opportunities of mathematics in digital times: Preschool teachers’ views,” Education Sciences, vol. 12, no. 7, 2022. https://doi.org/10.3390/educsci12070459 [35] K. M. M. Naser, S. H. Halili, and R. A. Razak, “Students’ acceptance of using a flipped STEM classroom Engineering-based module,” JuKu: Jurnal Kurikulum & Pengajaran Asia Pasifik, vol. 9, pp. 20–28, 2021. [36] T. Låg and R. G. Sæle, "Does the flipped classroom improve student learning and satisfaction? A systematic review and meta-analysis," AERA Open, vol. 5, no. 3, pp. 2332858419870489, 2019. https://doi.org/10.1177/2332858419870489 [37] K.-Y. Lee and Y.-C. Lai, “Facilitating higher-order thinking with the flipped classroom model: a student teacher’s experience in a Hong Kong secondary school,” Res. Pract. Technol. Enhanc. Learn., vol. 12, no. 1, p. 8, 2017. https://doi.org/10.1186/s41039-017- 0048-6 [38] H.-T. Hung, “Clickers in the flipped classroom: bring your own device (BYOD) to promote student learning,” Interact. Learn. Environ., vol. 25, no. 8, pp. 983–995, 2017. https://doi. org/10.1080/10494820.2016.1240090 [39] K. Umam, T. Nusantara, I. N. Parta, E. Hidayanto, and H. Mulyono, “An application of flipped classroom in mathematics teacher education programme,” Int. J. Interact. Mob. Technol., vol. 13, no. 03, p. 68, 2019. https://doi.org/10.3991/ijim.v13i03.10207 [40] J. O’Flaherty, C. Phillips, S. Karanicolas, C. Snelling, and T. Winning, “Corrigendum to ‘The use of flipped classrooms in higher education: A scoping review’ [The Internet and Higher Education 25 (2015) 85–95],” Internet High. Educ., vol. 27, p. 90, 2015. https://doi.org/10.1016/j.iheduc.2015.02.002 [41] M. Kim, E. Jung, A. de Siqueira, and L. Huber, “An investigation into effective pedagogies in a flipped classroom: A case study,” International Journal of E-Learning & Distance Education, vol. 31, no. 2, p. 1, 2016. [42] S. Yorganci, “Implementing flipped learning approach based on ‘first principles of instruction’ in mathematics courses,” Journal of Computer Assisted Learning, vol. 36, no. 5, pp. 763–779, 2020. https://doi.org/10.1111/jcal.12448 iJIM ‒ Vol. 17, No. 10, 2023 27 https://doi.org/10.1080/09500693.2016.1262567 https://doi.org/10.1007/s10956-020-09823-x https://doi.org/10.2307/j.ctt6wq7bh.7 https://doi.org/10.7771/2157-9288.1129 https://doi.org/10.1080/14926%E2%80%93156.2017.1380868 https://doi.org/10.1080/14926%E2%80%93156.2017.1380868 https://doi.org/10.3390/educsci12070459 https://doi.org/10.1177/2332858419870489 https://doi.org/10.1186/s41039-017-0048-6 https://doi.org/10.1186/s41039-017-0048-6 https://doi.org/10.1080/10494820.2016.1240090 https://doi.org/10.1080/10494820.2016.1240090 https://doi.org/10.3991/ijim.v13i03.10207 https://doi.org/10.1016/j.iheduc.2015.02.002 https://doi.org/10.1111/jcal.12448 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach [43] C. K. Lo, K. F. Hew, and K. F. Hew, “Using ‘first principles of instruction’ to design mathematics flipped classroom for underperforming students,” International Journal of Learning and Teaching, vol. 3, no. 2, pp. 82–89, 2017. [44] Y. Song, M. Jong, M. Chang, and W. Chen, “Guest Editorial: ‘HOW’ to Design, Implement and Evaluate the Flipped Classroom? -A Synthesis,” Educational Technology & Society, vol. 20, no. 1, pp. 180–183, 2017, Accessed: Sep. 21, 2019. [Online]. Available: http://maiga.athabascau.ca/publication/Editorial-2017-ETS.pdf [45] B. S. Barrett, A. L. Moran, and J. E. Woods, “Meteorology meets engineering: an interdisciplinary STEM module for middle and early secondary school students,” Int. J. STEM Educ., vol. 1, no. 1, p. 6, 2014. https://doi.org/10.1186/2196-7822-1-6 [46] E. Ababneh, A. Al-Tweissi, and K. Abulibdeh, “TIMSS and PISA impact – the case of Jordan,” Res. Pap. Educ., vol. 31, no. 5, pp. 542–555, 2016. https://doi.org/10.1080/ 02671522.2016.1225350 [47] M. Jamil, S. Ridhuan Mohd, M. Said, and K. Ibrahim, “Kompetensi guru terhadap pengurusan pengajaran dan pembelajaran: Suatu pendekatan teknik fuzzy delphi,” JuPiDi: Jurnal Kepimpinan Pendidikan, vol. 1, no. 3, pp. 77–88, 2017. [48] C. K. Lo and K. F. Hew, “Using ‘first principles of instruction’ to design secondary school mathematics flipped classroom: The findings of two exploratory studies,” Journal of Educational Technology & Society, vol. 20, no. 1, pp. 222–236, 2017. [49] A. Collin and R. A. Young, The Future of Career. 2000. https://doi.org/10.1604/ 9780521640213 [50] A. M. Akhmedovna, “What Flipped Classroom Is And How It Works,” The American Journal of Social Science and Education Innovations, vol. 03, no. 04, pp. 635–638, Apr. 2021, https://doi.org/10.37547/tajssei/volume03issue04-103 [51] A. Alhazmi, A. Imtiaz, F. Alhammadi, and E. Kaed, “Success and failure aspects of LMS in E-learning systems,” Int. J. Interact. Mob. Technol., vol. 15, no. 11, p. 133, 2021. https://doi. org/10.3991/ijim.v15i11.20805 [52] X. Cao and L. Yu, “Exploring the influence of excessive social media use at work: A three- dimension usage perspective,” Int. J. Inf. Manage., vol. 46, pp. 83–92, 2019. https://doi.org/ 10.1016/j.ijinfomgt.2018.11.019 [53] Ismail, I. Mahazir, Othman, Amiruddin, and Ariffin, “The use of animation video in teaching to enhance the imagination and visualization of student in engineering drawing,” IOP Conf. Ser. Mater. Sci. Eng., vol. 203, p. 012023, 2017. https://doi.org/10.1088/1757-899X/203/1/ 012023 [54] M. Ibrahim, P. D. Antonenko, C. M. Greenwood, and D. Wheeler, “Effects of segmenting, signalling, and weeding on learning from educational video,” Learn. Media Technol., vol. 37, no. 3, pp. 220–235, 2012. https://doi.org/10.1080/17439884.2011.585993 [55] J. Kämpfe, P. Sedlmeier, and F. Renkewitz, “The impact of background music on adult listeners: A meta-analysis,” Psychol. Music, vol. 39, no. 4, pp. 424–448, 2011. https://doi. org/10.1177/0305735610376261 [56] P. J. Guo, J. Kim, and R. Rubin, “How video production affects student engagement: An empirical study of MOOC videos,” in Proceedings of the First ACM Conference on Learning @ scale conference, 2014. [57] Lee, Kin-yuen, and Yiu-chi Lai. “Facilitating higher-order thinking with the flipped classroom model: a student teacher’s experience in a Hong Kong secondary school." Research and Practice in Technology Enhanced Learning, vol. 12, no. 1, p. 1-14, 2017. https://doi.org/10.1186/s41039-017-0048-6 [58] K. K. Szpunar, H. G. Jing, and D. L. Schacter, “Overcoming overconfidence in learning from video-recorded lectures: Implications of interpolated testing for online education,” J. 28 http://www.i-jim.org http://maiga.athabascau.ca/publication/Editorial-2017-ETS.pdf https://doi.org/10.1186/2196-7822-1-6 https://doi.org/10.1080/02671522.2016.1225350 https://doi.org/10.1080/02671522.2016.1225350 https://doi.org/10.1604/9780521640213 https://doi.org/10.1604/9780521640213 https://doi.org/10.37547/tajssei/volume03issue04-103 https://doi.org/10.3991/ijim.v15i11.20805 https://doi.org/10.3991/ijim.v15i11.20805 https://doi.org/10.1016/j.ijinfomgt.2018.11.019 https://doi.org/10.1016/j.ijinfomgt.2018.11.019 https://doi.org/10.1088/1757-899X/203/1/012023 https://doi.org/10.1088/1757-899X/203/1/012023 https://doi.org/10.1080/17439884.2011.585993 https://doi.org/10.1177/0305735610376261 https://doi.org/10.1177/0305735610376261 https://doi.org/10.1186/s41039-017-0048-6 Paper—Designing of a Flipped STEM Classroom Engineering-Based Module: Fuzzy Delphi Approach Appl. Res. Mem. Cogn., vol. 3, no. 3, pp. 161–164, 2014. https://doi.org/10.1016/j.jarmac. 2014.02.001 [59] C. Nantha, P. Pimdee, and J. Sitthiworachart, “A quasi-experimental evaluation of classes using traditional methods, problem-based learning, and flipped learning to enhance Thai student-teacher problem-solving skills and academic achievement,” Int. J. Emerg. Technol. Learn., vol. 17, no. 14, pp. 20–38, 2022. https://doi.org/10.3991/ijet.v17i14.30903 [60] T. Long, J. Cummins, and M. Waugh, “Use of the flipped classroom instructional model in higher education: instructors’ perspectives,” Journal of Computing in Higher Education, vol. 29, no. 2, pp. 179–200, Aug. 2016. https://doi.org/10.1007/s12528-016-9119-8 11 Authors Khadijeh Naser is an Assistant Professor at the Faculty of Education at Al Ain University, UAE. She participated in setting quality standards and performance indicators for schools in Jordan. Her research interests include Instructional Design & Technology, STEM Education, and Flipped Learning (Email: Khadijehmohammad 78@gmail.com). Sherin Alamassi is an Assistant Professor at the Faculty of Education at Al Ain University, UAE. She has 22 years of experience in professional development, organisational excellence, and integrating technology in learning and teaching (Email:sherin.abushamon@gmail.com). Zuhrieh Shana is an Associate Professor at the Faculty of Education at Al-Ain University, UAE. She possesses over 20 years of teaching, consulting, training, and research experience in different academic institutions in the USA, Saudi Arabia, Canada, and the United Arab Emirates (Email: zoeshanaa@yahoo.com). Jihan Yousef is an Assistant Professor at Al Ain University, teaching and supervising undergraduate and diploma students at the College of Education. She focused on improving teachers’ teaching skills and student learning throughout her career as a principal (Email: Jihan.Mousa@aau.ac.ae). Siti Hajar Halili is an Associate Professor at the Faculty of Education at the University of Malaya, Kuala Lumpur. Her research interests include Instructional Design & Technology, Open Learning Design & Technology, and Instructional Systems & Technology (Email: siti_hajar@um.edu.my). Article submitted 2023-01-20. Resubmitted 2023-03-16. Final acceptance 2023-03-23. Final version published as submitted by the authors. iJIM ‒ Vol. 17, No. 10, 2023 29 https://doi.org/10.1016/j.jarmac.2014.02.001 https://doi.org/10.1016/j.jarmac.2014.02.001 https://doi.org/10.3991/ijet.v17i14.30903 https://doi.org/10.1007/s12528-016-9119-8 mailto:Khadijehmohammad78@gmail.com mailto:Khadijehmohammad78@gmail.com mailto:sherin.abushamon@gmail.com mailto:zoeshanaa@yahoo.com mailto:Jihan.Mousa@aau.ac.ae mailto:siti_hajar@um.edu.my