Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4, 1841-1855 2024 Publisher: Learning Gate DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate © 2024 by the author; licensee Learning Gate * Correspondence: siti_hajar@um.edu.my Designing the open-badges guideline based on the fuzzy Delphi method Siti Hajar Halili1*, Norharyanti Mohsin2, Zamzami Zainuddin3, Juliana Othman4, Shukri Sulaiman5 1,2,3,4Faculty of Education, Universiti Malaya, Kuala Lumpur; siti_hajar@um.edu.my (S.H.H.) yantimohsin1@um.edu.my (N.M.) zamzami@um.edu.my (Z.Z.) julianaothman1@um.edu.my (J.O.). 5School of Distance Education, Universiti Sains Malaysia, Penang; shukriss@usm.edu.my (S.S.). Abstract: This study aims to obtain expert consensus on the appropriate elements of open badge learning guidelines. The Fuzzy Delphi Method (FDM) was used to reach a consensus from 20 experts with different backgrounds. These guidelines were designed and developed based on the Fuzzy Delphi Method (FDM) as well as the integration of the ASSURE instructional design model by Heinich, Molenda, and Russell (1993). The guidelines were evaluated by selected experts using several criteria, and data were collected through questionnaires. Based on the results of the study, all items meet the requirements in the number of three fuzzy triangles and pass the de-fuzzing process (more than Alpha α – cut-off value 0.5). Therefore, the findings of the study show that the experts agree that this module is suitable to be implemented according to the six themes set. The findings highlight key constructs for OBL guideline development included several key constructs related to OBL guideline content, state guideline objectives, method, media/tool, and material selection, as well as recommendations for use and participation based on the ASSURE instructional design model. Further research can incorporate insights from fields such as psychology, sociology, and educational policy. The focus should be on technical aspects and use qualitative and quantitative methods to ensure that all students, regardless of background, receive fair qualifications and recognition. Keywords: Fuzzy Delphi Method, Instructional design, Open education, Open-badges learning. 1. Introduction The landscape of distance education has undergone rapid change along with advances in information, communication, and technology (ICT) in this digital age. Open Distance Learning (ODL), formerly known as Distance Education (DE) since the early 1980s, is now one of the fastest-growing forms of global education. The merging of the concepts of "open learning" and "distance education" in the term "open and distance learning" emphasizes the importance of providing unlimited access to education as well as flexible learning opportunities, which eliminate time and location restrictions. The main principle of open and distance learning is to promote an educational system that is "open," which supports the sustainability of distance education practices. In theory, strategies for open and distance learning offer the possibility of learning with the separation of time or space between teachers and students, or a combination of both, and are implemented by various institutions or agencies. In the context of ODL, it is important to integrate the views and expectations of both sides, namely teachers and students, because today's students are more dependent on technology than previous generations. According to (1) the hope is that as technology advances, education will also evolve by leveraging technology to enhance the learning process. Apart from planning content and imparting knowledge, teachers are also expected to play a role as teaching designers, guiding and facilitating students throughout their education process (2; 3). 1842 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate 2. Literature Review Open badge learning (OBL) has been integrated as an innovative pedagogical approach in ODL, reflecting a modern educational model that offers alternative ways to increase the standard of instruction and learning. This model extends the learning experience beyond the confines of the traditional classroom, allowing students to control their educational journey and develop at their own pace. Besides focusing on providing access to technology or tools for students, it also explores innovative teaching methods where technology benefits both teachers and students in the learning environment (4). OBL is an innovative pedagogical approach to display students' knowledge, skills, and achievements, which are appropriate and competitive in the Industrial Revolution 4.0 (IR4.0) environment. Although the concept of open learning has grown rapidly in recent years, research on how to apply this new learning approach in the education system is still limited (5;6). Research that investigates the changing educational needs of students in the context of open badge learning, especially in Malaysia, is still lacking. Therefore, (7) emphasized the need for more studies on open learning, especially in the aspects of evaluation and learning experiences, because it is very necessary to meet the demands of the workforce in the era of the Industrial Revolution 4.0. In Malaysia, previous studies on open education focused more on MOOCs (8; 9). Although there are guidelines for micro-qualification practices issued by the Malaysian Qualifications Agency (MQA) in 2019 and 2020, no specific guidelines have been introduced by MQA for open learning. In addition, so far, there are no models, modules, guidelines, or frameworks that study in-depth open learning that can be used as a reference by higher education institutions to build and implement open badges in their programs. Therefore, this study aims to address those shortcomings by designing guidelines that help practitioners understand the concept of open learning as a future qualification that is flexible and accessible online. (10) stated that open badges have been used to guide learning, increase engagement, and encourage learning experiences, both in the context of skills, knowledge, and formal or informal learning activities. Therefore, according to (11) as well as (12), more research is needed in open education, especially in aspects such as OBL design, assessment, and learning experiences. This is important because alternative qualifications are essential in the workforce to meet the demands of Industrial Revolution 4.0 (13; 14). Thus, the purpose of this research is to determine the constructs and elements in designing the OBL guideline. The research questions guiding the study are ‘What are the constructs and elements in designing the OBL guideline?’ 3. Method 3.1. Research Design Based on their opinions regarding the appropriateness of the suggested guideline, an analysis is conducted. Therefore, this study uses the fuzzy Delphi method to gather expert opinions that invalidate the guideline to design and develop it during this phase. The Delphi technique and fuzzy set theory are combined to create fuzzy Delphi (15). (16) developed this analytical approach to decision- making, which combines fuzzy theory with the conventional Delphi method. Three features contribute to the Delphi technique, an expert opinion survey method: statistical group response, iteration and controlled feedback, and anonymous response (17). As per (18), the approach facilitates the amalgamation of viewpoints obtained autonomously from every specialist via repeated questionnaire cycles for forecast results. To put it succinctly, the fuzzy Delphi method is employed to ascertain the expert consensus acting as respondents through the application of quantitative methods. 1843 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate 3.2. Sampling Through purposive sampling, a panel of experts were selected for this phase to provide their consensus on the guideline. The process of choosing the appropriate sample for the FDM is not a non- probability sampling (19). This method was used because the samples are chosen based on their expertise and background in the subject matter, rather than at random. Choosing a suitable panel of experts is the most crucial step in the Delphi method because it influences the quality of the study's output (20). (21) states that lecturers with more than five years of experience are considered experts because they have continuous experience in both managing and instructing. Apart from that, according to (22) a person with more than five years of experience in a particular field is considered an expert in the field of education. The expert panel was chosen based on several criteria, including the presence of professors or senior lecturers with more than ten years of experience working with technology and communication in a variety of academic fields as researchers or educators. According to (23), a panel of five to ten experts is necessary for the study to meet its specific goals. However, according to (24), to ensure thorough and trustworthy research findings, a typical number of experts selected is between 15 and 35 experts. Given the high importance of consistency, multiple experts ranging from 10 to 50 are advised (25). Taking into account the relevant variables, a panel of 20 experts from diverse academic fields will be formed to assess and validate the model during this Fuzzy Delphi phase. They will be chosen to acquire knowledge and consensus from experts regarding the rationale behind the creation of guidelines. The evaluation is predicated on answers to a professional survey with a 7-language Likert scale. The ASSURE instructional design model serves as another foundation for the guideline. In 1999, (26) created the ASSURE model. This served as a manual for creating lessons and providing instruction using instructional technology. The ASSURE acronym stands for these important components: A- Analyze Learners, S- State Objectives, S- Select Instructional Methods, Media/Tools, and Materials, U- Utilize Media/Tools and Materials, R- Require Learner Participation, and E- Evaluate. Since it represents one of the instructional designs in a more structured method of instruction in the teaching and learning process, the ASSURE model was used in this study (27). This explains why this model was chosen for the study because it meets the objective of creating the guideline efficiently and methodically. 3.3. Instruments The instrument is organised into two sections: (a) background demographics and (b) content for creating the guidelines. It is created using the results of needs analyses, interviews with experts, and systematic literature reviews. Part A is the demographic information with 7 questions and Part B is the design content guideline consisting of 6 parts based on ASSURE instructional design model and of 55 questions with 7 Likert scales (1-Very Strongly Disagree, 2- Strongly Disagree, 3- Disagree, 4- Neither Agree nor Disagree, 5- Agree, 6- Strongly Agree and 7- Very Strongly Agree). 3.3.1. Validity of the Instruments Three experts participated to validate the instruments for the FDM survey (design) and interview (development). Table 1 shows the list of the experts involved in this study. 1844 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate Table 1. Background of experts for validity of instruments. Experts Post Expertise 1 Lecturer Expert in language & e-learning 2 Lecturer Expert in curriculum & instructional technology 3 Instructional designer Apps and system development 3.4. Data Collection and Analysis This study uses the FDM template to analyze the data, which has been developed by (28) experts in FDM analysis. Data from all items were analyzed based on the level of agreement between experts, where the threshold value (d) was ascertained by measuring the difference between two fuzzy numbers in the following approach: The expert agreement was evaluated using defuzzification values a priority score was assigned to each item. Every item was examined, and the study's findings showed that the experts reached an agreement, with a threshold value (d) exceeding 0.2 and the probability of agreement reaching or exceeding 75%. To measure expert agreement on each item on the guideline has a priority score based on its defuzzification value. The following are the actions suggested by (28) and (29): 1. As indicated in Table 2, an expert is asked to assess the significance of the evaluation criteria for the variables assessed using linguistic variables. Table 1. Likert Scale and Fuzzy Scale Table 2. Representation of Likert scale and fuzzy scale. Linguistic variables Fuzzy scale 1 0.9,1,1 2 0.7,0.9,1 3 0.5,0.7,0.9 4 0.3,0.5,0.7 5 0.1,0.3,0.5 6 0,0.1,0.3 7 0,0,0.1 2. Every linguistic variable is transformed into fuzzy triangular numbers. Fuzzy numbers represent variables for each criterion for experts e.g. and 1845 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate 3. The distance between fuzzy numbers is calculated using the peak method (30) with the condition that the threshold value d (m, n) must be less than or equal to 0.2 (31) based on the formula: 4. If the group consensus percentage exceeds 75% (32;33), the following step can now be taken. Otherwise, it is best to apply the second round of the Fuzzy Delphi Method. 5. Fuzzy evaluation is combined using the appropriate formula: 6. For each alternative, defuzzing is performed using a specific formula, and the position of the alternative is determined based on the value obtained. Acceptance of expert consensus also requires an α cut-off rate of 0.5 or higher (34; 35) such as: The instrument was distributed in hardcopy forms and 20 experts from Malaysian higher learning institutions (government and private Universities), International institutions and industry were involved in this study. The list of experts participating in this study is presented in Table 3. Table 3. Background of experts for the validity of instruments. Experts Post Expertise Working experience 1 Lecturer Curriculum and instructional design 20 years 2 Lecturer Curriculum development & future studies 30 years 3 Lecturer Educational technology 20 years 4 Lecturer Educational technology 10 years 5 Lecturer TVET curriculum development 10 years 6 Lecturer Instructional technology 15 years 7 Lecturer Curriculum and instruction 10 years 8 Lecturer Open education & technology 15 years 9 Lecturer Instructional technology 15 years 10 Lecturer Curriculum & language 15 years 11 Lecturer Curriculum and instruction 30 years 12 School principal Curriculum and educational technology 30 years 13 Lecturer Science computer & educational technology 15 years 1846 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate Experts Post Expertise Working experience 14 Lecturer Curriculum and educational technology 10 years 15 Lecturer Instructional technology 10 years 16 Lecturer Distance learning 8 years 17 Lecturer Curriculum and e-learning development 25 years 18 Assistant director TVET curriculum 30 years 19 Industry professional Training industry 20 years 20 Industry Professional Learning and development industry 15 years 4. Results The researcher has designed an initial instrument for the FDM assessment survey and shared it with 3 experts to validate the instrument before distributing it to 20 experts. 20 experts have been identified and have analyzed data via the FDM template. The procedures outlined in (28) should be followed to calculate the fuzzy Delphi Method. such as selection of expert, selection of scale, findings mean, identifying ‘d-value- threshold value’, achieving 75% consensus, calculating fuzzy evaluation value and defuzzification. The report is based on the design and development in Phase 2 which includes FDM findings based on ASSURE model as follows: 4.1. Part 1: Analysis Requirement for Guideline Content In this section, the sub-components consist of the target focus and target audience. These components are important to identify the theme requirement to design the guideline content. 4.1.1. Target Focus For target focus, employing a threshold value calculation where d ≤ 0.2, 2 elements were found to be accepted which showed a consensus percentage of more than 75%. However, 2 elements were rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled such as suitable for communities and practitioners. The results are shown in Table 4. Table 4. Target focus. Target focus Sub-components Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Suitable for higher learning students 0.158 100 15.733 0.874 1 2 Suitable for communities 0.250 50 13.600 0.756 R 3 Suitable for practitioners 0.249 52 15.667 0.770 R 4 Suitable for trainers 0.174 80 15.567 0.865 2 4.2. Target content When calculating the threshold value for target content, all accepted components were kept when the value of d ≤ 0.2 of consensus percentage of more than 75% for the first rank which is education, 1847 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate instructional design, curriculum development and emerging technologies. However, the expert panel rejected the technical aspect. Table 5 tabulated the findings. Table 5. Target content. Target content Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Education 0.158 100 15.733 0.874 1 2 Curriculum development 0.165 95 15.367 0.854 3 3 Emerging technologies 0.184 94 15.300 0.850 4 4 Technical aspect 0.222 50 14.267 0.793 R 5 Instructional design 0.123 100 15.633 0.869 2 4.2. Part 2: State Guideline Objectives For state guideline objectives, threshold value analysis retaining d ≤ 0.2 as the remained value of 5 elements was accepted out of 6 elements and a consensus percentage of greater than 75%. The experts' panel determined that the most important component to have in the guideline would be to explore the understanding of the OBL concept. This was succeeded by a readiness to implement OBL and focus on OBL instructional mapping in the third rank. Theme element based on understanding the digital skills in OBL practice was ranked fourth and understanding the recognition of informal learning ranked fifth. One element was rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled which is to recognize the technical aspect of utilizing OBL. Table 6 tabulated the findings. Table 6. State guidelines objectives. State guidelines objectives Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Explore knowledge of the OBL concept 0.163 100 15.500 0.861 1 2 Readiness to implement OBL 0.168 100 15.267 0.848 2 3 Understand the recognition of informal learning 0.172 100 14.733 0.819 5 4 Understand the digital skills in OBL practice 0.192 100 14.767 0.820 4 1848 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate State guidelines objectives Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 5 Focus on OBL instructional mapping 0.168 100 15.000 0.833 3 6 Recognize the technical aspect of utilizing OBL 0.291 57 11.600 0.644 R 4.3 Part 3: Selection of Method, Media/Tools and Materials In this section, the sub-components consist of selection of method, selection of media/tools and selection of materials. These components are important to identify the theme requirement to select of method, media/tools and materials to design the guideline. 4.4. Selection of Method For the selection of method, threshold value analysis retaining d ≤ 0.2 as the remained value of 4 elements were accepted out of 6 elements and a consensus percentage of greater than 75%, has only demonstrated 4 elements accepted out of 5. The expert panels ranked game-based learning in the first rank, followed by microlearning and project-based learning in the third rank. Experiential learning is placed at fourth rank. One element was rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled which is social learning. Table 7 tabulated the findings. Table 7. Selection of method. Selection of Method Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzz y score 1 Game-based learning 0.165 100 15.133 0.841 1 2 Microlearning 0.166 100 14.600 0.811 2 3 Project-based learning 0.179 94 14.567 0.809 3 4 Experiential learning 0.159 94 14.533 0.807 4 5 Social learning 0.322 67 11.133 0.619 R 4.5. Selection of Media/Tools This element of selection of media/tools referred to any learning materials that should be included to design the OBL guideline. 8 of the elements were retained after applying the threshold value calculation with a value of d ≤ 0.2, and the second requirement with a consensus percentage of greater than 75% was satisfied. Mobile apps received the highest ranking from the expert panels, followed by 1849 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate Video Tutorials and Artificial Intelligence (AI) in the third rank. Augmented Reality (AR) and Virtual Reality (VR) are placed at fourth rank, followed by Podcasts and Audio Resources, Interactive Chatbots, Interactive Simulations and Personal Websites. One element was rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled which is Community Forums. Table 8 tabulated the findings. Table 8. State guidelines objectives. Selection of media/Tools Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Mobile apps 0.090 100 16.333 0.907 1 2 Personal websites 0.182 94 14.400 0.800 8 3 Interactive simulations 0.179 94 14.967 0.831 7 4 Artificial intelligence (AI) 0.102 95 15.700 0.883 3 5 Podcasts and audio resources 0.117 100 15.533 0.863 5 6 Augmented Reality (AR) and virtual reality (VR) 0.165 95 15.900 0.872 4 7 Video tutorials 0.114 100 15.900 0.899 2 8 Interactive chatbots 0.138 100 15.467 0.859 6 9 Community forums 0.221 50 13.467 0.748 R 4.6. Selection of Materials For the selection of materials, when d ≤ 0.2 in the threshold value calculation retained all the elements with a consensus percentage greater than 75%. The panel of experts agreed that Open Educational Resources (OER) to be placed in the first place. This is followed by Infographics and Visual Aids and Multimedia Integration. At the fourth rank are Case Studies followed by suitable E- books. However, the expert panel rejected the Online Resource Communities. Table 9 tabulated the findings. Table 9. Selection of materials. Selection of materials Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Case studies 0.192 100 14.767 0.820 4 2 Infographics and visual aids 0.168 100 15.267 0.848 2 3 E-books 0.172 100 14.733 0.819 5 4 Multimedia integration 0.168 100 15.000 0.833 3 1850 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate Selection of materials Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 5 Open educational resources (OER) 0.163 100 15.500 0.861 1 6 Online resource communities 0.244 45 12.533 0.696 R 4.6. Part 4: Utilize Method, Media/Tools and Materials For utilising method, media/tools and materials content, the threshold value calculation with the value of d ≤ 0.2 was kept, as well as the acceptance of four elements in the content with an agreement percentage of more than 75%. The fuzzy score ranked explore interactive demos and tutorials provided by open badge platforms in the first place. The second place was to discover how open badges can incorporate multimedia elements. It is followed by providing brief training sessions on digital literacy skills and sharing examples of digital badges earned by others to motivate learners. One element was rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled which is to collaborate in online communities or forums related to open badges. Table 10 tabulated the findings. Table 10. Utilize method, media/tools and materials. Utilize method, media/tools and materials Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Explore interactive demos and tutorials provided by open badge platforms 0.168 100 15.600 0.867 1 2 Discover how open badges can incorporate multimedia elements 0.170 94 15.467 0.859 2 3 Provide brief training sessions on digital literacy skills 0.170 94 15.467 0.859 3 4 Share examples of digital badges earned by others to motivate learners 0.165 94 15.367 0.854 4 5 Collaborate in online communities or forums related to open badge 0.291 50 11.733 0.652 R 1851 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate 4.7. Part 5: Require Participation For required participation, when d ≤ 0.2 in the threshold value calculation, all the components were retained. Additionally, the second requirement of a consensus percentage of more than 75% was met, accepting 8 out of 9 elements. Expert panels reached a consensus to curate an open-source project to be placed at the first rank. This is followed by facilitating peer collaboration and providing collaborative projects. At the fourth rank is assigned hands-on projects, followed by creating an e- portfolio, incorporating peer reviews and integrating the interactive online modules and surveys. The panel experts ranked conduct industry guest webinar at the last place. However, the expert panel rejected the participation in discussion forums. Table 11 tabulated the findings. Table 11. Require participation. Require participation Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Integrate the interactive online modules and surveys 0.147 100 15.200 0.844 7 2 Facilitate Peer collaboration 0.109 100 15.800 0.878 2 3 Provide collaborative projects 0.123 100 15.633 0.869 3 4 Assign hands-on projects 0.138 100 15.467 0.859 4 5 Curate open-source project 0.137 100 15.933 0.885 1 6 Incorporate peer reviews 0.127 100 15.267 0.848 6 7 Participate in discussion forums 0.259 45 8.667 0.481 R 8 Create E-portfolio 0.173 89 15.333 0.852 5 9 Conduct industry guest webinar 0.184 89 14.700 0.817 8 4.8. Part 6: Evaluation For evaluation, threshold value analysis retaining d ≤ 0.2 as the remained value of 4 elements was accepted out of 6 elements and a consensus percentage of greater than 75%, which is the second required. According to the consensus of experts, continuous feedback mechanisms rank first, followed by reflection and journaling. It is followed by open-ended assessments and interactive assessments. Two elements were rejected since the Triangular Fuzzy Numbers requirements weren't fulfilled which are self-assessment tools and badges certification exam. Table 12 tabulated the findings. 1852 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate Table 12. Evaluation. Evaluation Elements Triangular fuzzy numbers requirements Defuzzification process requirement Rank Threshold value (d) Consensus percentage (%) Fuzzy evaluation Fuzzy score 1 Open-ended assessments 0.178 94 14.833 0.865 3 2 Interactive assessments 0.174 100 15.100 0.839 4 3 Self-assessment tools 0.212 100 13.433 0.746 R 4 Badges certification exam 0.307 100 12.633 0.702 R 5 Continuous feedback mechanisms 0.155 94 15.633 0.869 1 6 Reflection and journaling 0.144 100 15.567 0.867 2 5. Discussion The FDM findings highlight key constructs for OBL guideline development. The summarized results included several key constructs related to OBL guideline content, state guideline objectives, method, media/tool, and material selection, as well as recommendations for use and participation according to the ASSURE model of instructional design (26). For example, the analysis emphasised the importance of guidelines tailored to the needs of higher education students and trainers, and also instructional design, curriculum development, and emerging technologies in OBL. In addition, the findings addressed objectives such as investigating OBL concepts, improving readiness for OBL implementation, and emphasising instructional mapping and digital skills. Additionally, recommendations were made for the selection and use of various methods, media/tools, and materials to enhance the OBL experience, such as game-based learning, mobile apps, video tutorials, and open educational resources. Furthermore, strategies for encouraging active participation, such as facilitating peer collaboration, developing e-portfolios, and incorporating interactive online modules, were discussed. Overall, the FDM findings provide insight into the design and development of OBL initiatives, providing a comprehensive framework for future implementation efforts. The findings recommended integration of diverse methods, media/tools, and materials can enrich the academic curriculum while also encouraging innovation and engagement in teaching practices. Learning institutions can use the study's findings to make strategic decisions about curriculum development and instructional design. (12) also agreed that the use of open badges able to motivate, scaffold and recognize leraning in education. The FDM findings offer a structured framework for developing OBL guideline that address the diverse needs of students and lecturers. This is because (36) believe that there are many opportunities and challenges with digital open badges for learning institutions Policymakers can take advantage of the findings of this study to formulate educational policies and initiatives that promote innovation and technology integration in higher education. FDM findings can be used to form guidelines and standards for the implementation of OBL initiatives in an educational context. By adapting policies to the needs and preferences of students and academics, policymakers can create an environment that supports the adoption of OBL practices and improves educational outcomes. Stakeholders, including industry partners, educational institutions, and community members, can benefit from the results of this study by taking an active role in the design and implementation of OBL initiatives. Through collaboration with academics, policymakers, and educational institutions, stakeholders can contribute to the development of innovative OBL practices that meet the changing needs of students and society. 1853 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate The results of this investigation also have significant implications for the ASSURE instructional design model. The use of FDM shows the importance of a thorough and systematic methodological approach in research design. This study increases the data’s validity by ensuring that the assessment instrument is validated by a panel of experts before use, which adds credibility to the study results. In addition, the structured approach in summarizing FDM findings emphasizes the importance of clear and concise data compilation and synthesis, facilitating interpretation and decision-making in the DDR process. Additionally, the findings are consistent with the principles of the ASSURE instructional design model, indicating that this model is useful in guiding the design and development of educational interventions such as the OBL initiative. This integration ensures that OBL initiatives are student- centred, use diverse teaching strategies, and align with learning objectives. Overall, this study emphasizes the importance of methodological rigour and transparency in the research process, in terms of how effectively instructional design model is employed such as ASSURE in guiding innovative development in education. 6. Conclusion The guidelines offer a comprehensive approach, leverage innovative technologies and emphasize the importance of collaboration, making them a useful resource for educators and students. OBL's guideline approach, tailored for students and educators in higher education, is praised for providing educationally relevant content, instructional design, and new technologies. Its objectives from exploring the concept of OBL to emphasizing digital skills were effectively achieved through a variety of methods and materials. The findings of this study are inline with study by (37) that stated it is vital to use digital badges for professional development in higher education. Future research may be more successful by involving a broader range of expertise and perspectives in analyzing these guidelines. Although the focus on instructional designers and industry professionals is very beneficial, it risks ignoring insights from other fields such as psychology, sociology, or educational policy. A study by (38) and (39) suggest that badges in social media be investigated from a social-psychological perspective as well as considering the impact of open badges in education. By combining insights from multiple disciplines, researchers can gain a more complete understanding of learning needs and qualifications. This inclusive approach can produce guidelines that are more comprehensive and able to meet the needs of different students and educators, as well as reduce the potential for bias due to a narrow focus on only one area. For example, views from underrepresented groups or fields, such as minority students or nontraditional educators, may offer important perspectives that can enrich the guideline development process and promote equity in eligibility. Future studies also need to focus on the technical aspects of the OBL guidelines. Although current guidelines have covered many aspects of pedagogical and instructional design, there may be shortcomings in addressing technical issues such as platform compatibility, data security, or badge issuance procedures. By focusing on these technical aspects, researchers can ensure that the OBL guidelines are not only pedagogically strong but also sound and technically applicable in various educational settings. This focus can help resolve implementation challenges and ensure the effective use of OBL initiatives. Another suggestion for future research is to investigate how the guideline development process can encourage fair and inclusive credentialing practices. The evaluation of these guidelines can be done using qualitative and quantitative methods to ensure that qualifications and recognition are equitably accessible to all students. Researchers can examine issues such as transparency of eligibility criteria, access to badge opportunities, and inclusivity in assessment methods. It can help develop a more inclusive and fair credentialing system by identifying barriers to fair credentialing practices and formulating strategies to overcome them. (40) also agreed that digital badges able to unlock 1854 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate transformative potential in education perspectives. This emphasis on fairness and inclusion is consistent with larger efforts to promote social justice and equity in education and qualifications. Funding: The authors would like to acknowledge the Agensi Kelayakan Malaysia for the financial support through the Agensi Kelayakan Malaysia (MQA) Grant - GA 010-2022. Copyright: © 2024 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). References [1] T. M. Alqahtani, F. D. Yusop, and S. H. Halili, "Dataset on the relationships between flipped classroom approach, students learning satisfaction and online learning anxiety in the context of Saudi Arabian higher education institutions," Data in Brief, vol. 45, Art. no. 108588, 2022. DOI: 10.1016/j.dib.2022.108588 [2] R. Jamaludin, E. McKay, and S. Ledger, "Are we ready for Education 4.0 within higher education institutions? Thriving for knowledge, industry and humanity in a dynamic higher education ecosystem?" Journal of Applied Research in Higher Education, vol. 12, no. 5, pp. 1161-1173, 2020. [3] A. R. Rafiza, K. Dalwinder, H. Siti, and R. Zahri, "Flipped ESL teacher professional development: Embracing change to remain relevant," Teaching English with Technology, vol. 16, no. 3, pp. 85-102, 2016. [4] S. H. Halili, H. Sulaiman, and A. R. Rafiza, "Information and communications technology acceptance among Malaysian adolescents in urban poverty," Turkish Online Journal of Educational Technology, vol. 16, pp. 47-54, 2017. [5] N. H. Ibrahim, M. N. Masran, and I. N. Abdul Rahim, "Student perception toward MOOC in study skills class," International Journal of Education and Pedagogy, vol. 2, no. 2, pp. 56-63, 2020. [6] A. R. Rafiza, F. D. Yusop, H. Siti, and C. S. Raman, "Electronic continuous professional development (E-CPD) for teachers: Bridging the gap between knowledge and application," Turkish Online Journal of Educational Technology, vol. 14, no. 4, pp. 14-27, 2015. [7] N. Che Ahmat, A. Ahmad Ridzuan, and M. Yunos, "Perceptions and readiness of educators toward micro-credential certification programme," International Journal of Education and Pedagogy, vol. 4, no. 1, pp. 38-50, 2022. [8] J. A. Kumar and H. Al-Samarraie, "An investigation of novice pre-university students’ views towards MOOCs: The case of Malaysia," The Reference Librarian, vol. 60, no. 2, pp. 134-147, 2019. [9] G. Santos-Hermosa, N. Ferran-Ferrer, and E. Abadal, "Repositories of open educational resources: An assessment of reuse and educational aspects," The International Review of Research in Open and Distributed Learning, vol. 18, no. 5, pp. 84-120, 2017. [10] B. Oliver, "Better 21st century credentials: Evaluating the promise, perils and disruptive potential of digital badges," Smart Learning Environments, vol. 3, no. 1, pp. 1-12, 2016. [11] K. Clements, R. E. West, and E. Hunsaker, "Getting started with open badges and open micro-credentials," The International Review of Research in Open and Distributed Learning, vol. 21, no. 1, pp. 153-171, 2020 [12] J. Jovanovic and V. Devedzic, "Open badges: Novel means to motivate, scaffold and recognize learning," Technology, Knowledge and Learning, vol. 20, pp. 115-122, 2014. [13] F. Nasreen, S. H. Halili, and R. A. Razak, "Employability skills of Malaysian university students for IR4.0: A systematic literature review," Malaysian Online Journal of Educational Management, vol. 10, no. 4, pp. 15-28, 2022. [14] C. L. Lim, P. K. Nair, M. J. Keppell, N. Hassan, and E. Ayub, "Developing a framework for the university-wide implementation of micro-credentials and digital badges: A case study from a Malaysian private university," in 2018 IEEE 4th International Conference on Computer and Communications (ICCC), 2018, pp. 1715-1719. [15] T. J. Murray, L. L. Pipino, and J. P. Gigch, "A pilot study of fuzzy set modification of Delphi," Human Systems Management, vol. 5, no. 1, pp. 76-80, 1985. [16] A. Kaufman and M. M. Gupta, Fuzzy Mathematical Models in Engineering and Management Science. Amsterdam: Elsevier, 1988. [17] C. C. Hsu and B. A. Sandford, "The Delphi technique: Making sense of consensus," Practical Assessment, Research & Evaluation, vol. 12, no. 10, pp. 1-8, 2010. [18] N. R. M. Nurul, S. Saedah, and S. H. Siti, "Aplikasi teknik fuzzy Delphi terhadap keperluan elemen teknologi sebagai wadah dalam pembelajaran berasaskan pemikiran rekabentuk," Asia Pacific Journal of Educators and Education, vol. 34, pp. 129-151, 2019. https://creativecommons.org/licenses/by/4.0/ 1855 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 1841-1855, 2024 DOI: 10.55214/25768484.v8i4.1556 © 2024 by the authors; licensee Learning Gate [19] F. Hasson, S. Keeney, and H. McKenna, "Research guidelines for the Delphi survey technique," Journal of Advanced Nursing, vol. 32, no. 4, pp. 1008-1015, 2000. DOI: 10.1046/j.1365-2648.2000.t01-1-01567.x [20] R. Taylor, L. Judd, S. Witt, and L. Moutinho, "Delphi method applied to tourism," Tourism Marketing and Management Handbook, pp. 95-98, 1989. [21] D. C. Berliner, "Describing the behaviour and documenting the accomplishments of expert teachers," Bulletin of Science, Technology & Society, vol. 24, no. 3, pp. 200-212, 2004. [22] R. Akbari and E. Yazdanmehr, "A critical analysis of the selection criteria of expert teachers in ELT," Theory and Practice in Language Studies, vol. 4, no. 8, pp. 1653-1658, 2014. [23] H. A. Linstone and M. Turoff, The Delphi Method: Techniques and Applications. Addison-Wesley Publishing Company, 2002. [24] T. J. Gordon, "The Delphi method," in Futures Research Methodology, 3rd ed., J. C. Glenn and T. J. Gordon, Eds. The Millennium Project, 2009. [25] K. Jones and B. Twiss, "Using a panel of experts to validate instructional objectives," Educational Technology Publications, 1978. [26] R. Heinich, M. Molenda, J. D. Russell, and S. E. Smaldino, Instructional Media and Technologies for Learning. Upper Saddle River, NJ: Prentice-Hall, 1999. [27] C. T. Tan and S. R. Siti, "Enhancing higher-order thinking skills through a panel of experts: A case study," Journal of Educational Research and Practice, vol. 5, no. 2, pp. 34-45, 2015. [28] R. M. J. Mohd, S. Saedah, H. Zaharah, M. N. Nurulrabihah, and A. A. Ahmad, Pengenalan Asas Kaedah Fuzzy Delphi dalam Penyelidikan RekaBentuk dan Pembangunan. Bangi, Selangor: Minda Intelek Agency, 2017. [29] P. T. Chang, L. C. Huang, and H. J. Lin, "The fuzzy Delphi method via fuzzy statistics and membership function fitting and an application to the human resources," Fuzzy Sets and Systems, vol. 112, pp. 511-520, 2000. [30] Chen, "Extensions of the topics for group decision-making under fuzzy environment," Fuzzy Sets and Systems, vol. 114, no. 1, pp. 1-9, 2000. [31] C.-H. Cheng and Y. Lin, "O.R. applications: Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation," European Journal of Operational Research, vol. 142, pp. 174-186, 2002. [32] H.-C. Chu and G.-J. Hwang, "A Delphi-based approach to developing expert systems with the cooperation of multiple experts," Expert Systems with Applications, vol. 34, pp. 2826-2840, 2008. [33] J. W. Murry and J. O. Hammons, "Delphi: A versatile methodology for conducting qualitative research," Qualitative Research, vol. 18, no. 4, pp. 423-436, 2017. [34] S. Bodjanova, "Median alpha-levels of a fuzzy number," Fuzzy Sets and Systems, vol. 157, pp. 879-891, 2006. [35] C.-W. Tang and C.-T. Wu, "Obtaining a picture of undergraduate education quality: A voice from inside the university," Higher Education, vol. 60, no. 3, pp. 269-286, 2010. [36] T. Farmer and R. E. West, "Opportunities and challenges with digital open badges," Educational Technology, vol. 56, no. 5, pp. 45-48, 2016. [37] P. Dyjur and G. Lindstrom, "Perceptions and uses of digital badges for professional learning development in higher education," TechTrends, vol. 61, no. 4, pp. 386-392, 2017. [38] J. Ahn, A. Pellicone, and B. S. Butler, "Open badges for education: What are the implications at the intersection of open systems and badging?" Research in Learning Technology, vol. 2, pp. 1-13, 2014. [39] J. Antin and E. Churchill, "Badges in social media: A social psychological perspective," in Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 2011, pp. 1-10. [40] D. Thomson and P. Wolstencroft, "Digital badges: Unlocking their transformative potential," Journal of Education for Teaching, vol. 45, no. 4, pp. 484-495, 2019.