Microsoft Word - 11-3751_s_ETASR_V10_N5_pp6209-6213 Engineering, Technology & Applied Science Research Vol. 10, No. 5, 2020, 6209-6213 6209 www.etasr.com Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering Programs: A Case Study Sidi M. Ahmed Ghaly Electrical Engineering Department College of Engineering Al Imam Mohammad Ibn Saud Islamic University Riyadh, Saudi Arabia smghaly@imamu.edu.sa Abstract—Outcome and objective assessment is a method for determining whether students and graduates have learned, have retained, and can apply what they have been taught. Assessment plans have to include a statement of educational objectives, measurements of attainment of the objectives, and use of the assessment results for continuous improvement. In this paper, the experience of the Electrical Engineering program at Al Imam Mohammad Ibn Saud Islamic University is used as an example of how an indirect assessment method is established and implemented. Many surveys have been used and contributed to the numerical analysis for the indirect assessment of the program outcomes and achievements. The indirect assessment results indicate that the target levels are achieved for all the seven program outcomes and the three objectives that have been considered. Keywords-engineering; education assessment; learning outcomes; educational objectives; accreditation; assessment loop; ABET I. INTRODUCTION The last few years have witnessed a growing recognition of the need for assessment and evaluation of the learning outcome [1-2]. These are directed not only at large-scale, standardized tests but also at classroom assessment practices. At least two factors have contributed to the demands for assessment and evaluation: the changing nature of educational objectives and the relationship between assessment, teaching, and learning. Assessment is defined as one or more processes that identify, collect, and prepare the data necessary for evaluation. Evaluation is defined as one or more processes for interpreting the data acquired through the assessment processes to determine how well the program objectives are attained. The Electrical Engineering (EE) Department at Al Imam Mohammad Ibn Saud Islamic University (IMSIU) has been implementing different assessment and evaluation approaches [3-5] for learning outcomes and Program Educational Objectives (PEOs), the concept of which is adopted to meet the international and national accreditation requirements. This paper describes an indirect method used to assess and evaluate the achievement of program outcomes and objectives. Student Outcomes (SOs) or program outcomes describe what students are expected to know and be able to do by the time of graduation while PEOs describe what graduates are expected to know and be able to do after graduation. The EE program is very keen on developing and implementing high quality education. For this reason, it requests periodic input from its constituents [6]. To improve the curriculum and the overall improvement of instructional delivery, review and evaluation of the program outcomes and objectives are conducted in each academic cycle. This principal feedback loop affects curriculum organization and content. Also, faculty members provide a collection of feedback forms that are specific to a course or relating to the entire program. The EE department receives feedback through these forms to encourage an unencumbered exchange of ideas and responses to the material presented in class and to the overall curriculum design. This is particularly helpful for students to be able to receive and benefit from current and most up to date content and knowledge. The EE program periodically consults the industry, to ensure that the department’s efforts align well with its needs [7]. The EE program educational objectives serve its constituencies. The program constituencies are any entity or organizations that benefit from the outcomes of the program. The current EE educational objectives require periodic revisions to reflect the needs of the program constituencies in greater depth. The EE department keeps in contact with current and prospective employers of its graduates. Such constituents are strongly encouraged to join the industrial affiliates program which provides valuable feedback to the department faculty members for student preparation. When necessary, appropriate actions take place through the advising unit and/or the Quality Committee. Exit survey is very important in program outcomes evaluation. Alumni and employer surveys also provide important feedback regarding employment opportunities for students, and the opportunity to assess and evaluate the program educational objectives. Although many studies have been published on the direct assessment methods of different engineering programs [8-10], the indirect assessment approach does not receive much attention. In this study, the measurement of SOs and PEOs achievements provides considerable information on the effectiveness of an EE program seeking to meet the accreditation criterion [11]. Corresponding author: Sidi M. Ahmed Ghaly Engineering, Technology & Applied Science Research Vol. 10, No. 5, 2020, 6209-6213 6210 www.etasr.com Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … II. BACKGROUND A. Program Outecomes or Student Outcomes The EE program has adopted the new SOs for the engineering program as prescribed in Accreditation Board for Engineering and Technology (ABET) Criterion 3 [12]. The abilities that students must demonstrate at the time of graduation are attained through various courses taken by all students during the program, which are: 1. Identify, formulate, and solve complex engineering problems by applying principles of engineering, science, and mathematics. 2. Apply engineering design to produce solutions that meet specified needs with consideration of public health, safety, and welfare, as well as global, cultural, social, environmental, and economic factors. 3. Communicate effectively with a range of audiences. 4. Recognize ethical and professional responsibilities in engineering situations and make informed judgments, which must consider the impact of engineering solutions in global, economic, environmental, and societal contexts. 5. Function effectively on a team whose members together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks, and meet objectives 6. Develop and conduct appropriate experimentation, analyze and interpret data, and use engineering judgment to draw conclusions. 7. Acquire and apply new knowledge as needed, using appropriate learning strategies. B. Program Educational Objectives Graduates of the EE Program should attain at least one of the following educational objectives after graduation: • PEO1: Serve competently the needs of industry and academia by demonstrating high-quality knowledge, research, and skills in the area of EE. • PEO2: Pursue professional development through professional study and self-learning with full gratitude of the importance of professional and ethical responsibility. • PEO3: Contribute to the welfare of society through the responsible practice of engineering, leadership, and teamwork. C. Consistency of the PEOs with the Mission of the Institution The following statements summarize the University, College, and Department missions: • Mission1: High-quality Education (University/College/ Department). • Mission 2: Conduct innovative research/leadership in scientific research/ (University/College/Department) • Mission 3: Local and global community service/offer professional services (University/College/Department). • Mission 4: Equip students with the skills to be life-long contributors to their profession (Department). Table I shows the mapping between the Institution’s missions and the PEOs indicating the consistency of the PEOs with the mission of the institution. TABLE I. PEOS MAPPING TO THE MISSIONS OF THE INSTITUTION Mission 1 Mission 2 Mission 3 Mission 4 PEO 1 X X X PEO 2 X X PEO 3 X X D. Relationship of SOs to PEOs SOs support the PEOs. The first objective (PEO1) is supported by most of the SOs (1, 2, 5, and 6). The second objective (PEO2) is supported by 4, 5, and 7 SOs. The third objective (PEO3) is supported by SOs 4, 5, and 7. Table II summarizes the mapping between the SOs and the PEOs. TABLE II. SOS TO PEOS MAPPING PEOs SOs 1 2 3 1 X 2 X 3 X 4 X X 5 X X X 6 X 7 X X III. INDIRECT ASSESMENT Indirect assessment and evaluation at the EE program were conducted the academic years 2017-18 and 2018-19 (two cycles). The frequency of data collection and review process for the indirect assessment approach of SOs and PEOs are summarized in Table III. TABLE III. DATA COLLECTION FREQUENCY AND REVIEW PROCESS Assessment tool Frequency of assessment Expected level of attainment Exit survey Every year 70% Industrial advisory survey Every year 70% Alumni survey Every three years 70% Employer survey Every three years 70% Faculty survey Every year 70% Student survey Every year 70% The EE program has approved a benchmark level of outcome and objective achievement (two thresholds) for the indirect assessment. These thresholds are an average score of 20% (Unsatisfactory) and 70% (Satisfactory). For an outcome or objective to be considered achieved, the indirect assessment should reach an average score of at least 70% [13]. This should be considered as evidence illustrating that the level of the student outcome and program objective achievement is satisfactory. Engineering, Technology & Applied Science Research Vol. 10, No. 5, 2020, 6209-6213 6211 www.etasr.com Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … IV. RESULTS OF THE EVALUATION PROCESS In this section, the results and evaluation process are presented. At first the process in cycle 2 is discussed and then a comparison is performed between the results from cycles 1 and 2. To evaluate the results obtained from the surveys, the performance targets were defined to assess the level of attainment. Table III summarizes the tools for the indirect assessment and evaluation of program outcomes and objectives. The surveys listed in Table III, are being used in the assessment process at various stages. These surveys provide feedback on the weaknesses and strengths of the SOs and the PEOs. The exit survey contributes to the measurement of the indirect assessment of student outcomes 1-7 and provides the program with useful information to aid the development of an action plan for continuous improvement. A. SO Assesment through the Exit Survey At the end of each semester, the EE program carries out a survey of its graduating students. The exit survey contributes to the measurement of the indirect assessment of student outcomes 1-7 and provides useful information that aids the development of an action plan for the continuous improvement of the program. In this survey, it is useful to ask students to rate the program learning outcomes or SOs in order to indicate the extent of their achievement. The total number of respondents for both cycles is 40 students. The survey questions are related to the specific outcomes 1-7 using the same key performance indicators (KPIs) that are used in direct assessment [14-16] as seen as example for the outcome 1 in Table IV. The data collected from the students' evaluation are stored and analyzed electronically and the results are considered for further evaluation and taking appropriate actions. TABLE IV. TYPICAL INDIRECT ASSESSMENT FROM SO 1 SO 1 Level KPI Attribute question 5 4 3 2 1 1.1 Your ability to apply mathematical and scientific principles to formulate models and systems relevant to electrical engineering. 1.2 Your ability to solve electrical engineering problems by using the concepts of integral and differential calculus and/or linear algebra. 1.3 Your ability to execute calculations correctly 1.4 Your ability to translate academic theory into engineering applications. 1.5 Your ability to apply analytical, graphical or numerical methods to solve complex problems 1.6 Your ability to solve practical engineering problems logically using correct theoretical concepts. 1.7 Your ability to formulate alternative solutions to a complex engineering problem. 1.8 Your ability to identify the governing concept of a complex engineering problem. The results obtained for the outcome 1 are summarized in Figure 1. This Figure is shown as an example of the average obtained for each KPI of the outcome 1 from the survey questions on a scale of 5 is very satisfied down to 1 for very dissatisfied and then changed to percentage and shown graphically. Figure 2 represents the exit survey results of cycle 2 (fall 2018 and spring 2019) based on all program outcome assessment data (1 through 7). Fig. 1. KPIs assessment results–outcome 1. Fig. 2. Indirect assessment results for all program outcomes. The achievement is considered satisfactory if the weighted average is greater than or equal to 70%. As seen, all program outcomes meet the program expectations. B. Comparison of Assessment Results The comparison of the direct assessments' results of the program outcomes (SOs 1-7) for both cycles (2017-18 and 2018-19) is depicted numerically in Table V and graphically in Figure 3, where the satisfactory results for cycles 1 and 2 are shown in blue and red respectively. TABLE V. COMPARISON OF THE ASSESSMENTS’ RESULTS Student Outcomes (SOs) Achievements 1 2 3 4 5 6 7 Cycle 1 2017-18 84 80 84 75 89 76 87 Cycle 2 2018-19 78 84 85 81 85 84 82 It can be observed that the program outcome attainment for both cycles is well above the satisfaction target. However, some attainments in cycle 1 have degraded to some extent as compared to cycle 2. This issue was investigated in the early part of the 2019-2020 fall and the faculty’s opinion was sought to suggest measures for improvement. Engineering, Technology & Applied Science Research Vol. 10, No. 5, 2020, 6209-6213 6212 www.etasr.com Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … Fig. 3. Comparison of the two cycles of indirect assessment results. C. SO and Objective Assessment from the Alumni Survey During both cycles, the EE Department conducted a survey of alumni to assess their knowledge, skills, and perceptions of relevant aspects of their college experiences in order to evaluate the program objectives. The alumni were asked many questions, including SOs and program objective achievement. The presented data are based on a random sample of 13 alumni. Again the achievement is considered satisfactory if the weighted average is greater than or equal to 70%. The results are shown in Table VI and it can be observed that program outcomes and objectives indicate satisfaction. If a weighted average was less than 70% for an SO, it would be discussed in the EE Quality Committee to analyze the reason(s) behind it. The matter might also be escalated to the Department Council for remedial measures to be taken. TABLE VI. TYPICAL INDIRECT ASSESMENT ALUMNI FORM My Education at IMSIU has given me the ability to: 5 4 3 2 1 Averag e % # Attribute question 1 Apply disciplinary reasoning, critical thinking, and hands-on skills to analyze, design, and solve engineering problems. 3 5 4 1 0 61 2 Consider professional, ethical, and social responsibility of engineering technology practices. 5 5 3 0 0 77 3 Communicate effectively. 0 10 2 1 0 77 4 Perform effectively, think independently, and work collaboratively in a team environment. 5 6 2 0 0 84 5 Participate in professional development, including continuous self-improvement and lifelong learning. 5 6 2 0 0 84 6 PEO1 6 5 1 1 1 84 7 PEO2 2 8 2 1 0 77 8 PEO3 4 6 3 0 0 77 D. SO and Objective Assessment from the Employer Survey The employer survey is also assessed at intervals of 3 years. A set of randomly selected employers are requested to fill a questionnaire available online. The employer was asked many questions, including SO and program objective achievement. The presented data are based on a sample of 6 employers. The same satisfaction criterion of 70% is applied. The results are shown in Table VII and it can be observed that all program outcomes indicate satisfaction. TABLE VII. TYPICAL INDIRECT ASSESMENT EMPLOYER FORM The following section of the questionnaire is intended to seek your feedback on “Demonstrated Abilities of our Graduates” while they are employed at your company: 5 4 3 2 1 Average % # Attribute question 1 Apply knowledge of mathematics, science, and engineering. 3 2 1 0 0 83 2 Design and conduct experiments. 5 1 0 0 0 100 3 Solve engineering problem, analyze and interpret data. 4 2 0 0 0 100 4 Design a system, component, or process to meet desired needs. 1 3 2 0 0 66 5 Understand professional and ethical responsibilities. 0 5 1 0 0 83 6 Communicate effectively. 3 3 0 0 0 100 7 Understand the impact of engineering solutions in a global and societal context. 2 4 0 0 0 100 8 Recognize the need to engage in lifelong learning. 1 3 2 0 0 66 9 Use techniques, skills, and modern engineering tools necessary for engineering practice. 1 3 2 0 0 66 10 Serve competently the needs of industry and academia by demonstrating high-quality knowledge, research, and skills in the area of Electrical Engineering 3 3 0 0 0 100 11 Pursue professional development through professional study and self-learning with full gratitude of the importance of professional and ethical responsibility 2 3 1 0 0 83 12 Contribute to the welfare of society through the responsible practice of engineering, leadership, and teamwork 3 2 0 1 0 83 E. SO Assessment through the Industrial Advisory Board (IAB) and the Senior Students Table VIII shows the response of the IAB to the question “to what extent you are satisfied with the formulation of the EE PEOs?” TABLE VIII. IAB SURVEY FOR PEOS To what extent you are satisfied with the formulation of the EE PEOs? PEO1 PEO2 PEO3 Very satisfied 4 1 3 2 Satisfied 3 4 3 2 Neutral 2 1 0 1 Dissatisfied 1 0 0 1 Very dissatisfied 0 0 0 0 Result 93% 100% 68% Also, a total of 23 students (seniors) responded to the contacted survey. These 23 students are all in level 7 and 8 classes. The students were presented with the formulation of PEOs along with a brief explanation for each PEO. The results and of the survey are shown in Table IX. Engineering, Technology & Applied Science Research Vol. 10, No. 5, 2020, 6209-6213 6213 www.etasr.com Ghaly: Indirect Evaluation of Program Educational Objectives and Student Outcomes for Engineering … TABLE IX. SENIOR STUDENT SURVEY RESULTS Senior student survey results EE PEO PEO1 PEO2 PEO3 Very satisfied 4 7 (30%) 8 (35)% 12 (50%) Satisfied 3 8 (40%) 6 (25%) 8 (35%) Neutral 2 7 (30%) 7 (30%) 2 (10%) Dissatisfied 1 0% 1 (5%) 1 (5%) Very dissatisfied 0 0% 1(5%) 0% More than 65% of the surveyed EE senior students are either satisfied or very satisfied with the EE’s PEOs V. CONCLUSION In this work, various surveys were used and contributed to the indirect assessment of the program outcomes and objectives attainment. The results of the exit survey for the last two cycles show that students’ perspective on their attainment of the program outcomes is satisfactory, since all the student outcomes are above the minimum required level of attainment. A meeting with the students is arranged every semester to clarify the objectives and the content of the indirect assessment. Additional surveys, such as the employer survey and the alumni survey are assessed every three years to get feedback from the employers and the alumni in the market about the program objectives and at which level they met their carrier requirements. REFERENCES [1] M. Alsuwaiket, A. H. Blasi, and R. A. Al-Msie’deen, “Formulating Module Assessment for Improved Academic Performance Predictability in Higher Education,” Engineering, Technology & Applied Science Research, vol. 9, no. 3, pp. 4287–4291, Jun. 2019. [2] P. Shayan and E. Iscioglu, “An Assessment of Students’ Satisfaction Level from Learning Management Systems: Case Study of Payamnoor and Farhangian Universities,” Engineering, Technology & Applied Science Research, vol. 7, no. 4, pp. 1874–1878, Aug. 2017. [3] S. M. Ahmed Ghaly, “Optimization of Quality Assessment and Evaluation Approach for Engineering Program Accreditation,” in 2019 IEEE Global Engineering Education Conference (EDUCON), Apr. 2019, pp. 1267–1274, doi: 10.1109/EDUCON.2019.8725162. [4] H. Al-Nashash et al., “Improving electrical engineering education at the American University of Sharjah through continuous assessment,” European Journal of Engineering Education, vol. 34, no. 1, pp. 15–28, Mar. 2009, doi: 10.1080/03043790802710169. [5] Y. Bai and R. Pigott, “Assessing Outcomes Using Program Assessment Portfolio Approach,” Journal of Professional Issues in Engineering Education and Practice, vol. 130, no. 4, pp. 246–254, Oct. 2004, doi: 10.1061/(ASCE)1052-3928(2004)130:4(246). [6] C. Crosthwaite, I. Cameron, P. Lant, and J. Litster, “Balancing Curriculum Processes and Content in a Project Centred Curriculum: In Pursuit of Graduate Attributes,” Education for Chemical Engineers, vol. 1, no. 1, pp. 39–48, Jan. 2006, doi: 10.1205/ece.05002. [7] M. Besterfield-Sacre et al., “Defining the outcomes: a framework for EC-2000,” IEEE Transactions on Education, vol. 43, no. 2, pp. 100– 110, May 2000, doi: 10.1109/13.848060. [8] J. B. Biggs and C. Tang, Teaching for Quality Learning at University: What the Student Does, 4th ed. UK: McGraw-Hill Education, 2011. [9] I. Warren, “Teaching Patterns and Software Design.,” presented at the Seventh Australasian Computing Education Conference (ACE 2005), Newcastle, Australia, Feb. 2005. [10] L. Alzubaidi, “Program Outcomes Assessment Using Key Performance Indicators,” in Proceedings of 62 nd ISERD International Conference, Boston, USA, Jan. 2017. [11] Y. Kalaani and R. Haddad, “Continuous Improvement in the Assessment Process of Engineering Programs,” Proceedings of the 2014 American Society for Engineering Education Southeast Section Conference, Mar. 2014, [Online]. Available: https://digitalcommons.georgiasouthern.edu/ electrical-eng-facpubs/43. [12] R. Petrova, A. Tibrewal, and T. Sobh, “An Electronic Web-based Assessment System,” Journal of STEM Education, vol. 7, no. 3, Dec. 2006. [13] L. Alzubaidi, I. Jabur, and R. Alzubaidi, “Course Assessment Tool Using the Combination Approach,” in Proceedings of the World Congress on Engineering and Computer Science 2015, Oct. 2015, vol. 1. [14] B. Ramasamy, “Attainment of Course Outcomes in Outcome Based Education: A Case Study,” Journal of Engineering Education Transformations, vol. 30, no. 2, pp. 82–85, Oct. 2016, doi: 10.16920/ jeet/2016/v30i2/105449. [15] Y. Abosalem, “Assessment Techniques and Students’ Higher-Order Thinking Skills,” International Journal of Secondary Education, vol. 4, no. 1, pp. 1–11, Mar. 2016, doi: 10.11648/j.ijsedu.20160401.11. [16] J. McGourty, C. Sebastian, and W. Swart, “Performance measurement and continuous improvement of undergraduate engineering education systems,” in FIE 1997, 1997, vol. 3, pp. 1294–1301, doi: 10.1109/ FIE.1997.632658.