Australasian Journal of Educational Technology, 2021, 37(6). 88 Developing and investigating an electronic performance support system (EPSS) for academic performance Baris Sezer Hacettepe University, Turkey The aim of this study was to develop an electronic performance support system (EPSS) to improve students’ performance in the topic of scientific research methods. A review of the literature shows that EPSSs are usually developed using advanced programming technologies. This limits the widespread use of EPSS for educators and researchers. In this study, a mid-level EPSS was developed by integrating Moodle to a static (non-interactive) educational system, without using advanced programming. The study was designed and carried out using the developmental research method. Study participants were second year students from the Faculty of Medicine of Hacettepe University in Turkey. The system was designed by taking students’ needs into account, as well as other issues that the literature identified as points of consideration when developing an EPSS. The mid-level EPSS developed within the framework of this study increased the students’ academic achievement and students stated that the system met their expectations. Implications for practice or policy: • Educators can support scientific research methods courses with EPSSs. • Educators and researchers may take advantage of detailed steps in this research, if they want to develop such an EPSS. • A mid-level EPSS is taken as reference when developing an EPSS without advanced programming skills. Keywords: developmental research, academic performance, learning management systems, scientific research methods, instructional design Introduction A rapid increase in the amount of knowledge and technological advancement means that static knowledge structures are now being replaced by easy-to-update dynamic structures. Dynamic structures can play an important role in carrying the process of knowledge acquisition or performance improvement from a starting point to a target level in a systematic manner. Rich information states and/or advanced skill sets are required to reach the target level more easily. Various sources of knowledge and skills (printed materials, apprenticeship, simulation, virtual learning environments, massive open online courses, electronic performance support systems etc.) can be used to meet these requirements (Carvalho, 2018). This study examines the use of an electronic peformance support system (EPSS), one of the knowledge sources listed above. The concept of EPSSs, coined by Gloria Gery in her 1991 book of the same title, has since undergone rapid development. Although many different definitions exist, an EPSS is defined by Reiser (2012), at a most basic level, as an integrated electronic environment with the goal of providing instant and easy access to information, advice, and experiences to improve individuals’ performance. In recent years, the EPSS has become one of the most effective means of improving human performance and learning (Askar, 2018; Carvalho, 2018; Kert et al., 2014; Li & Peng, 2016; Martinez-Mediano & Losada, 2017). Studies in the literature show that EPSSs improves users’ knowledge and skills (Erdogmus, 2015; Kert et al., 2014; Martinez-Mediano & Losada, 2017; Miller et al., 2007; Mitchell, 2014; Mitchem et al., 2016). Of the four basic components that an EPSS must have, the database can be described as a field where all information in the system is stored, the instruction system as a field where various types of information (video, visual, written) facilitating the use of EPSS are presented, the consultation system as the component that supports task-related performance and provides guidance, and the ancillary tools as various applications or tools to support performance improvement in line with user needs (Askar, 2018; McKay & Wager, 2007). An EPSS must have these four components, however, the relevant literature also identifies the following situations that need to be taken into account when developing an EPSS.: a simple design Australasian Journal of Educational Technology, 2021, 37(6). 89 (Askar, 2018; Barker et al., 2007; Bollen et al., 2015; Carvalho, 2018; Chen et al., 2009; Kalota & Hung, 2013b); brief subject-specific information (Barker et al., 2007; Bollen et al., 2015; Klein & Reiser, 2014; Martinez-Mediano & Losada, 2017); development on the basis of target audience analysis (Alparslan, 2009; Bollen et al., 2015; Chen et al., 2009; Kalota & Hung, 2013b); ease of updating (Bollen et al., 2015); development in the form of modular information structures (Bollen et al., 2015; Martinez-Mediano & Losada, 2017); provision of step-by-step task instructions (Li & Peng, 2016; Mitchell, 2014); use in real world job environments (Alparslan, 2009; Chen et al., 2009; Li & Peng, 2016; Martinez-Mediano & Losada, 2017; Mitchell, 2014); integration with learning management systems (Bollen et al., 2015; Dias et al., 2017; Martinez-Mediano & Losada, 2017; Mitchell, 2014); and interaction with other sources of information (Bollen et al., 2015; Chen et al., 2009; Mitchem et al., 2013; Ugur-Erdogmus & Cagiltay, 2018). Three approaches are taken as reference when developing an EPSS (Nguyen, 2007): • Minimal EPSS: These systems only include the basic characteristics (custom development, exclusion of irrelevant information, and operating in a computer environment) that an EPSS must have. These EPSS systems are usually developed as static structures that are not updated very frequently. • Mid-level EPSS: These systems are expected to provide the user with sufficient support to achieve a task (offering step-by-step task instructions, being easy-to-update, providing rapid access to information, providing functional help, being accessible whenever and wherever the task is to be achieved), rather than optimum support (providing different levels of information, taking different learning styles into account, incorporating artificial intelligence). • High-level EPSS: These full-featured systems provide optimum support for the user to achieve a task (incorporating artificial intelligence, presenting information in different ways, allowing a high level of user-content interaction). Integrated Tutors, designed and built for a compact, modular simulator for use in austere environments (no simulation labs, classrooms, debriefing facilities/equipment, Internet access or instructors), developed by Lizdas et al. (2017) is an example of this type of EPSS. The above conditions would be taken into consideration, depending on design principles of the EPSS to be developed, current resources, limitations, and the characteristics of target users. A small scale EPSS can be designed with less effort, but this is not the case with larger scale projects, which require models that would meet systematic planning needs of the design (Kalota & Hung, 2013a). One model of instructional design was the D4M2 model of EPSS design and development (Brown, 1996). The D4M2 stands for define, design, develop, deliver, and manage-measure. Brown (1996) redefined and expanded this process into a book on EPSS design, and called the new model the ED4 (define, design, develop, deliver): • Define phase: Analysis of the performer and the performance environment. The aim is to assess the performer’s needs, define the EPSS goals and requirements and plan the development process. • Design phase: Create storyboard or flowcharts of the detailed design. Important part of this step is to make the EPSS user friendly, including attention to suggestions made by the potential users. • Develop phase: Includes the development and testing of a functional prototype. When the prototype is satisfactory, it can be turned over to the systems developers. • Deliver and evaluation phase: After development, the EPSS has to be rolled out so that the performers can use it. This might necessitate installation procedures, training, notification of procedure changes, and other items that were planned for during the design phase. The literature shows that EPSSs developed in many studies (Bollen et al., 2015; Kert et al., 2014; Li & Peng, 2016; Lizdas et al., 2017; Martinez-Mediano & Losada, 2017; Mitchem et al., 2016) use advanced programming, particularly to achieve interaction and cooperation. Programming is a separate field of expertise, and educators cannot be expected to have this skill. Creating static educational websites, on the other hand, is a relatively simple task. It is common practice today to create simple and effective educational websites using ready-made web templates. In addition, learning management systems (LMS) play an important role in higher education, to help learners during the learning process. As an example, Moodle is a free, widely used LMS that supports online learning based on a social constructionist framework of education (Costello, 2013; Dias et al., 2017; Sezer & Yilmaz, 2019). It is argued that EPSSs can be Australasian Journal of Educational Technology, 2021, 37(6). 90 strengthened by integrating them with LMSs (Dias et al., 2017; Martinez-Mediano & Losada, 2017; Mitchell, 2014). Bollen et al. (2015) developed an EPSS integrated with Moodle. In a study conducted to improve college students’ performance in a psychology course, they report that students found the system to be practical, and that they made frequent use of the system for academic tasks. These positive findings aside, Bollen et al. recommended that further studies should be conducted to examine the effects of Moodle- integrated EPSSs on task performance and learning. Similarly, Kalota and Hung (2013b) argued that evaluation of the effects of EPSSs should be based on academic achievement and an assessment of user performance in addition to user satisfaction, as reflected in the products generated by users using the system. The purpose of the study and research question The above information shows that EPSSs can be effective tools for performance improvement and knowledge gain of individuals. The literature shows that EPSSs use advanced programming (Bollen et al., 2015; Kert et al., 2014; Li & Peng, 2016; Lizdas et al., 2017; Martinez-Mediano & Losada, 2017; Mitchem et al., 2016). This limits the widespread use of EPSSs due to programming as a separate field of expertise. Educators and researchers face difficulties for developing an EPSS when they do not have competence in this field. To address these issues, this study developed an EPSS without the need for programming. It was then integrated with Moodle to increase its potential use and capability. Given the context and current conditions, the decision was made that developing a mid-level EPSS would be more practical and appropriate for the needs of the target audience whose performance is to be improved. The system was designed on the basis of the ED4 model (Brown, 1996), the phases of which have been summarised in the previous section. The main aim of this study was to develop a mid-level EPSS to improve students’ performance in the topic of scientific research methods. The research question was: Is it possible to develop an EPSS using mid- level technologies that is effective or as effective as an EPSS created using high-level technologies? Methods This study was planned and performed as developmental research. Developmental research has been defined as the systematic study of designing, developing, and evaluating instructional programs, processes, and products that must meet the criteria of internal consistency and effectiveness (Richey et al., 2004). The generic development process was performed by analysis, design, development, implementation, and evaluation. The EPSS was used in the Good Medical Practice (GMP) program of the Faculty of Medicine of Hacettepe University, and its effects were examined. The GMP program consists of seven different modules. These modules are as follows: interprofessional collaboration, communication, clinical visits, ethics/professionalism, medical humanities, evidence-based medicine, and physical examination training. The program started in 2005. The goal of this program is for students to achieve the skills and attitudes needed by a good physician, who is competent in caring for and communicating with patients. During this study, the medical humanities (MH) module (the topic of scientific research methods) was held according to students’ needs. The ultimate goad of this module, is for students is to deliver a presentation at the medical humanities conference, held at the end of the semester. Participants The study participants were second year undergraduate students from the Faculty of Medicine of Hacettepe University in Turkey. The Faculty of Medicine had 506 secondyear students in the academic year 2018– 2019. These students were an homogeneous sample and were allocated into groups of 16 to 17 for the GMP program. These groups were formed by distributing equal numbers of students to groups in terms of gender, age, whether they attend the English or the Turkish medical program, and whether they were retaking the course. In total, 30 groups were formed, and this study was conducted with 9 of the groups (n = 152). Of the 152 participants, 47.4% were male (n = 72) and 52.6% were female (n = 80). All participants gave their informed consent. Australasian Journal of Educational Technology, 2021, 37(6). 91 Data collection tools Academic achievement test To prepare the pre- and post-test in this study, an initial form consisting a mix of 14 open ended, true/false, and multiple choice questions was created. The assessment items of the MH module are: (1) prepare a literature review on history, belief, religion, archeology, architectural philosophy, sociology, ethics, law and anthropology under the title of human and medicine; (2) prepare a report on the chosen topic, to send a summary of this report to the medical student congress at the end of the semester; and (3) prepare and present effective oral or poster presentations at the medical humanities conference. The 14 questions created for this research were examined by two experts in measurement and evaluation, and one in educational technology, using a rubric based on course objectives, grammar, distractors, and course level. Four questions were removed from the test based on the input of these experts. Multiple choice and true/false questions on the achievement test were scored as correct (10 points) or not (0 points). An assessment rubric was created for the open ended questions, and responses were scored using this rubric. Open ended questions were scored between 0 and 10 points. The highest possible score on the test was therefore 100 points. The achievement test was administered to students as a pre-test before designing the EPSS, 4 weeks before the MH session. It was again administered as post-test 1 week after the MH session. Educational website evaluation scale Feedback on the EPSS was collected using the educational website evaluation scale developed by Henkoğlu and Mıhçı (2012). The scale is reported to be a valid and reliable measurement tool for all educational platforms (EPSS, dynamic or static website, etc.) (Henkoğlu & Mıhçı, 2012). The scale consists of 16 items in total, organised into four dimensions, each containing four items: usefulness, confirmation of expectations, satisfaction, and continuance intention. The Cronbach’s alpha reliability coefficient was reported by Henkoğlu and Mıhçı (2012) to be 0.91 for the overall scale, 0.83 for perceived usefulness, 0.85 for confirmation of expectations, 0.85 for satisfaction, and 0.76 for continuance intention. In this study, the Cronbach’s alpha reliability coefficient was found to be 0.89 for the overall scale, 0.87 for perceived usefulness, 0.89 for confirmation of expectations, 0.83 for satisfaction, and 0.80 for continuance intention. Delphi/expert evaluation/Semi-structured interview forms Three forms were created for this reseach: • Delphi form: for students to evaluate the level of difficulty of the GMP modules on a 7-point Likert scale from 1 (minimum) to 7 (maximum). • Expert evaluation form: for experts to evaluate the students’ assignments at the end of the MH sessions. This form was created by the author and finalised after receiving expert opinion. The evaluation form consisted of eight items scored on a 5-point Likert scale, and based on criteria to evaluate academic abstracts from a scientific point of view. • Semi-structured interview forms: Semi-structured interviews were conducted with the participants during and after implementation. The interviews were held face-to-face, 4 weeks prior to, during, and 1 week after the MH session. In these interviews, students’ needs, expectations concerning the design architecture of the EPSS and views on the evaluation of the EPSS software were obtained. Data analysis The SPSS 22 program was used for statistical analysis of quantitative data collected for the research. The descriptive statistics were frequency valued, percent values, mean, and standard deviation. These statistics were used to analyse the quantitative data collected at the different stages of the study. Paired sample t-test was used to test if there were differences in achievement before and after the EPSS was introduced. Regarding student perceptions of the designed system, responses from three open-ended questions were reported in the deliver and evaluation phase. Students’ perceptions were collected to obtain feedback on the system were analysed by three experts separately, and categories were created. Data are presented both in tables, and as direct quotations in the section on findings. Results This section presents and comments on findings from the analysis of the collected data. Findings are presented in sub-sections following the phases of the ED4 model (Brown, 1996), which formed the basis of EPSS development in this study. Australasian Journal of Educational Technology, 2021, 37(6). 92 Define phase In the Faculty of Medicine, where the study was conducted, the GMP program is carried out intensively throughout the academic year. To select the module that would be the subject of the EPSS to be designed, a needs analysis was conducted. Needs were identified using the delphi method. After the first (n = 152) and second (n = 119) round it was found that the MH module was rated the lowest, that is, where students needed the most support. Therefore, there was no need to conduct a further delphi round. At this phase of the study, a focus group was held with 12 students to identify the students’ needs in more detail. In the meetings held with these students, issues in the MH module with which the students had the most difficulty, and possible solutions were discussed. Their views and suggestions, regarding how the EPSS could be used to address the problems they reported were collected. These headings were classified as: selection of the study topic, preparing and making a presentation, scientific research methods, and reporting. Headings to be used as inputs in the design phase of the EPSS (support topics) and potential solutions were then created. Before developing an EPSS, the software and hardware requirements of the system must be defined (Barker et al., 2007). Internet-based systems with dynamic or semi-dynamic structures are usually recommended (Bollen et al. 2015; Martinez-Mediano & Losada, 2017). As previously mentioned, EPSSs in the literature are frequently created using advanced programming technologies. In this study, a static educational website was created without advanced programming and was integrated with an LMS (Moodle) to improve its capability and potential uses (communication, chat, forum, file sharing, online assessments, etc.). The system was developed to be suitable for different platforms (tablet, smart phone, desktop, etc.). All information was shared with students in a subsequent focus group meeting, from which analysis report document was created, and the design phase was initiated. Design phase In this phase, work was conducted on the requirements for developing the EPSS. All of the conditions that need to be taken into account when developing an EPSS, as discussed in the introduction section, were given consideration. A flowchart incorporating these conditions was created. On the basis of the outputs of the analysis phase and recommendations on which components to include, the EPSS to be developed was designed in the form of modules including: communication, what is medical humanities, subject of study, scientific research, scientific reporting, preparing a presentation, frequently asked questions, access to moodle, and notifications. By taking user profiles and needs into account, the EPSS was designed in one semester, in the simplest manner possible, to provide rapid solutions to potential problems encountered. To this end, focus group meetings with 12 students were also held at the design phase. Depending on the frequency of the actual students’ questions, a decision was made to keep the frequently asked questions (FAQs) module up-to-date. In addition, a communication module, which provided e-mail access to the site administrator and a chat function between students, was added to facilitate updates and provide rapid feedback in case of system problems. Advisor and help sections, both of which are indispensable in an EPSS, were included in each module. For example, the module on selecting a study topic provides recommendations to students on which subjects would be appropriate topics, for example, information on the titles and contents of submissions accepted for oral presentation in MH conferences during the previous 9 years. It also provided a step-by-step explanation of how to plan a research project. Development phase The system was designed and developed using the HTML5 framework. The system was integrated with Moodle to increase usage capability and impact, for example, by allowing students to submit conference applications, communicate with each other using the chat function, forum, and submit assignments. The author created and provided information on the flowcharts, infrastructure, and database of the system. The system developed was frequently updated by the system administrator (author). Support was received from subject matter experts, as needed. The nature of this support was reviewing the content of the EPSS, Australasian Journal of Educational Technology, 2021, 37(6). 93 including types of studies and research design. The basic organisational chart for the system is summarised in Figure 1. Figure 1. Organisational chart of the system The system consisted of eight modules, with all modules containing an advisor-help section. 1. Communication/ask a question: Using Moodle, students could ask session, content, and system- related questions and receive answers, and interact with other students and the administrator. 2. Medical humanities: This module provided information on what the MH module involved, how the module proceeded over the semester, and what was expected from the students. 3. Subject of study: In this module, advice was provided on what to pay attention to for conference submissions to receive positive appraisals from reviewers at the end of the semester. Information was provided on what topics could be selected, what topics were inappropriate for the module, and what topics had been selected by students in previous years. All submissions accepted for oral presentation in the previous 9 years were presented in this module. 4. Scientific research: In this module, brief and clear information was provided on research designs, sampling processes, data collection tools, data collection methods, and data analysis. In the preparation of these content areas, support was received from an academic specialising in assessment and evaluation. Additional sources (32 online tutorials, tests, guided tours) were provided for students wanting more detailed information. 5. Scientific reporting and preparing a presentation: In this module, information was provided on what to pay attention to when submitting abstracts. Additional information and videos on how to prepare oral and poster presentations was also provided. 6. Frequently asked questions (FAQs): The 10 most frequently asked questions were featured in the FAQ module. In addition, some practical information was also presented to provide guidance to students. Australasian Journal of Educational Technology, 2021, 37(6). 94 7. Access to Moodle: In this module, students engaged in cooperative interaction using Moodle. Students were able to submit their module and system-related feedback, assignments, and project applications using this module. Students were given user names and passwords for accessing Moodle at the beginning of the semester, and help was provided to facilitate their use of the EPSS system. 8. Announcements: This module reminded students about MH module-related tasks, assignments, and conference application information and deadlines. Expert opinion was obtained from two faculty members specialising in educational technologies for initial evaluation of the system. At this phase, the software was checked manually for practicality and whether it had any broken links. The system was revised and finalised for implementation based on suggestions and recommendations received from the experts. A sample interface from the system is provided in Figure 2. Figure 2. The homepage of EPSS Deliver and evaluation phase The EPSS was uploaded to an intranet server. Unlike Moodle, no passwords were required to access the system. Students used the system for the real-world tasks tasks they needed to perform for the MH module. Subsequently, student learning was measured using the achievement test. Studies conducted by students were evaluated by experts. Student feedback on the system was received using the educational website Australasian Journal of Educational Technology, 2021, 37(6). 95 evaluation scale and interview forms. Findings from these three evaluations are presented below in their respective sections. Effects of the system on students’ academic achievement A paired sample t-test was conducted to see if the pre-test and post-test scores of the students who used the EPSS differed significantly. Pre-test was administered before designing the EPSS, and post-test was administered 1 week after the MH session. Results are reported in the Table 1. As Table 1 shows, a significant difference was found for the pre- and post-test academic achievement scores. Table 1 Comparison of pre-test and post-test scores using t-test Test N X SD t p Pre-test 74 36.75 9.56 25.91 .000 Post-test 74 80.02 10.08 Assessment of student products Independently of each other, two experts in medical education evaluated abstracts students submitted for their conference applications (n = 74). The results of these are presented Table 2., which shows that overall, the experts were of the opinion that studies by students met educational criteria. Table 2 Expert evaluation results (n = 74; range 1-5) Item Mean (Expert1) SD Mean (Expert2) SD Mean SD 1. Study topic is appropriate for medical humanities 4.53 0.51 4.51 0.67 4.52 0.59 2. Study title reflects contents 4.90 0.37 4.80 0.45 4.85 0.41 3. The method is appropriate 4.63 0.49 4.67 0.47 4.65 0.48 4. There is sufficient analysis and interpretation 4.52 0.51 4.50 0.50 4.51 0.51 5. There is sufficient discussion 4.21 0.53 4.25 0.62 4.23 0.57 6. Conclusion and/or recommendations are appropriate 4.31 0.58 4.33 0.76 4.32 0.67 7. The study makes an original contribution 4.57 0.50 4.55 0.76 4.56 0.63 8. The study follows style guidelines 4.78 0.41 4.72 0.45 4.75 0.43 Assessment of the system by students Feedback on the EPSS system was collected using the focus group form and evaluated using the educational website evaluation scale developed by Henkoğlu and Mıhçı (2012). Feedback was received from 74 students (37 males, 36 females, and 1 not stated) who made active use of the EPSS. Results reported in Table 3 show that, in the EPSS evaluation, students expressed positive views about all dimensions. This finding shows that EPSS met students’ expectations, they were satisfied with their use of the system, and they had a strong intention to use the EPSS again in the future. Australasian Journal of Educational Technology, 2021, 37(6). 96 Table 3 The scores of students’ evaluation of the EPSS ( X = mean, k = number of items, n = 74, range 1-5) Dimensions Min Max X X /k SD Usefulness 4.00 20.00 18.20 4.55 1.32 Confirmation of Expectations 4.00 20.00 18.50 4.62 1.27 Satisfaction 4.00 20.00 18.50 4.62 1.27 Continuance Intention 4.00 20.00 18.06 4.51 1.22 Total 16.00 80.00 73.26 4.57 1.34 Another focus group of 12 students (6 females and 6 males) was held, to obtain more detailed information. In this meeting, the author first held a brainstorming session with the participants and following this session data was collected by distributing interview forms consisting of three questions: (1) Please state the contribution of the system you have used to your learning, (2) Please indicate aspects of the system that require improvement, and (3) Would you like to use similar systems in the rest of your education? Form responses were analysed by three experts in each of measurement and evaluation, educational technologies, medical education. Findings from the analysis of student responses to the first item on the structured interview form are reported in Table 4. Student views on the effects of the EPSS on learning emphasised: (1) facilitation of the learning process because it provided them with clear and practical information and supported them with Moodle, (2) supported their acquisition of a professional perspective because it helped improve their academic writing skills, and (3) guided them to reliable additional sources. Table 4 Views on the effects of the system on student learning (n = 12) Categories f Facilitation 8 Professionalism 5 Guidance 4 Some of the student views and comments on this question were as follows. The MH module of Good Medical Practice was the module that I found to be the most difficult last year. The process felt very complex, and I felt like I wasn’t ready to conduct scientific research. The system that I used summarized everything very clearly. It was very useful. And also Moodle was very useful for manage (sic) my tasks. (Student A) The Frequently Asked Questions module had all the extra information I was looking for. In this way, I was able to complete my project without searching Google or other sources. (Student C) Last year, I wasn’t selected to make a presentation. I had serious shortcomings in terms of academic writing. I didn’t know how to express my ideas. This year, thanks to the system, my project was accepted. (Student F) Findings from the analysis of student responses to the second item on the interview form, “Please indicate aspects of the system that require improvement”, are reported in Table 5. Aspects of the EPSS that students viewed as requiring improvement, confirmed the expectation that the system should direct students to various sources or web pages based on learning analytics results, and present information in multiple formats based on their learning styles/preferences. Australasian Journal of Educational Technology, 2021, 37(6). 97 Table 5 Views on aspects of the system that require improvement (n= 12) Categories f Interaction 7 Presenting information in multiple formats 5 Some of the student views and comments on this question were as follows: When I entered the help menu, there was a user guide for the system. It was useful, but I wish it were in video format rather than in writing. It would have been more effective. (Student A) I wish there were pop-up warnings in the system, like they have in some education software. For example, when I read a topic, it could ask “Was this helpful? If not, you should check this topic.” Like Office Wizard. It would have been more impressive. (Student D) It would be better if answers in the Frequently Asked Questions module were in video format. (Student B) Findings from the analysis of student responses to the third item on the interview form, “Would you like to use similar systems in the rest of your education?” are reported in Table 6. Students stated that it would be nice to have a similar system for all their courses, particularly when they prepare for exams, as it would facilitate self-study and save them time and effort by providing them with clear information. Table 6 Views on the use of the system in other areas (n= 12) Categories f Exams 8 All courses 7 Some of the student views and comments on this last question were as follows: I wish this system were available for all courses, not just for GMP. I wish we could access information whenever we need. (Student F) It should definitely be used. This way, in our intensive study program, we could focus on modules that are the most difficult. (Student D) I wish there was a system like this when preparing for committee exams. It could summarize what to study and how. This way, we would be saving time. (Student A) An overall assessment of the feedback received in focus group meetings showed that students found the system to be very useful particularly because it provided clear information. In addition, second year student participants saw the EPSS as a very useful source of information on conducting scientific research, a topic with which they were not very familiar. On the other hand, students stated that the EPSS would make a bigger impact if it had a more interactive and personalised structure. In other words, the students’ expectations for the EPSS were high. In addition, students expected similar systems to be developed to cover all the courses in the curriculum. Discussion Previous studies (Alparslan, 2009; Bollen et al., 2015; Chen et al., 2009; Kalota & Hung, 2013b; Kert et al., 2014; Li & Peng, 2016; Lizdas et al., 2017; Martinez-Mediano & Losada, 2017; Mitchell, 2014; Mitchem et al., 2013) used advanced programming technologies to develop an EPSS. In this study, the Australasian Journal of Educational Technology, 2021, 37(6). 98 focus was on the development of a mid-level EPSS based on the ED4 model and providing interaction by integrating Moodle, while paying attention to recommendations in the literature concerning desirable EPSS features. The findings of this study show that academic achievement of students improved, and experts in medical education found the products that students created were adequate. This suggests that our EPSS was effective, and its contents met users’ needs. It is important for prospective physicians in the early phases of their academic careers to be able to manage the process of participating in a scientific conference, and know how to conduct an academic study. The system was able to provide effective support in these respects, which could be attributed to the direct involvement of students in the analysis and design phases of development of the EPSS, allowing effective collection of their needs and expectations. Significant improvements were made in the development phase of this study on the basis of suggestions and recommendations by educational technology experts, particularly concerning the interface and effective presentation of contents. The system was also revised in terms of content based on recommendations by two measurement and evaluation experts during the design and development phases. This shows the value of interdisciplinary cooperation in developing this kind of systems. The advisor–help section of the system, included in each module, and the FAQs module were updated on a regular basis, and it was a straightforward process. Answers to the main questions asked by students as they were using the EPSS were presented in the FAQs module and under the help menu, thus keeping the system dynamic. In the semi-structured interviews, students did not report encountering any technical problems in accessing and using the system. Studies show that encountering technical problems decreases satisfaction with, and frequency of, using an EPSS (Kalota & Hung, 2013a; Kert et al., 2014). However, in this study, students were involved in the design process, which allowed the author to make the system easy to use by intervening before the development phase to resolve issues that created difficulties for students. In addition, the help menu and the FAQs module of the system were kept up to date in quasi-real time, providing timely support to resolve difficulties students faced. Consistent with previous studies (Cleary & Zimmerman, 2012; Erdoğan, 2013; MacFadyen & Dawson, 2012; Raman et al., 2014), the integration of EPSS into Moodle in this study might have improved the usefulness of the system. This was similar to Bollen et al. (2015), who reported that students found a Moodle-integrated EPSS to be useful because the system was easy to use and its contents met their needs. In semi-structured interviews about the usefulness of the EPSS, students stated that the system was useful and effective. However, it was also found that students desired more interactive software in the from of an adaptive, personalised system to direct them to various information sources/pages on the basis of learning analytics. These expectations are consistent with the theoretical arguments in favour of EPSSs (Gery, 1991). Information should be provided to users in multiple formats. These are features that a more capable system, also known as a high-level EPSS, can offer and they are important for making the system more user-friendly and fit for its intended purpose (Nguyen, 2007). It is necessary to work with a team when developing a full- featured EPSS and, at the very least, this team should have a graphic designer, a software developer, an instructional designer, and a subject matter expert. However, because of the mid-level EPSS design, only one person (the author) created the EPSS for this research, with input from subject matter experts. This demonstrated that a viable EPSS could be created, disseminated, and used without need for advanced programming and still received overall positive student evaluations. Students also stated that the development of similar systems for all of their courses would improve their academic performance and learning. These findings show the benefits of the mid-level EPSS with the just- in-time feature that was developed in this study. This suggestion should be given serious consideration. Frequently mentioned together with the concept of just-in-time, these 24/7 systems provide instant access to essential information at critical moments and when students are receptive to learning. EPSS systems can contribute to students’ performance by assisting them in all their courses and in preparing for exams. Conclusion This study focused on the process of the development of an EPSS without using advanced programming for medical students for use in their GMP program, and the evaluation of the EPSS after use for real-world tasks. The mid-level EPSS developed within the framework of this study increased the students’ academic achievement in the topic of scientific research methods, and students stated that the system met their Australasian Journal of Educational Technology, 2021, 37(6). 99 expectations. They were satisfied with the performance of the system and would like to use similar systems in the future. The framework used in this study can provide guidance for future works. Limitations The study has some limitations. Firstly, this study was conducted with the participation of a limited number of students at Hacettepe University Medical School. Second, the effects of the EPSS on the academic achievement of students were not examined using a control group. This creates uncertainty as to whether the change in the dependent variable of academic achievement is due to EPSS use or some other factor. Recommendations to practitioners An educational technology centre could be established to develop EPSSs in universities. The developed EPSSs could be integrated into other faculties. Based on the previous years’ academic data, EPSSs could be developed for the most challenging courses. Multidisciplinary projects could be carried out to develop full-featured EPSSs, especially in conjunction with computer or software engineering faculties. Recommendations for further studies In future studies EPSSs should be evaluated in terms of their effects on technology acceptance and task completion. Nevertheless, regardless of the choice of environment, tool, or application, the most important point to consider is the quality of educational content and activities provided. The experiences of expert instructors, together with the needs of participants, should not be overlooked when preparing educational content. High-level EPSSs that use artificial intelligence and mid/minimal level-EPSSs could be compared in terms of their effects on student achievement and satisfaction. EPSSs with additional features could be tested for their effectiveness. Long-term follow-up studies need to be conducted to examine the effects of EPSSs on student behaviour in educational settings for academic writing. Future studies also need to be conducted with larger samples of students from different disciplines. References Alparslan, N. C. (2009). Development of an electronic performance support system for training people on radio laboratory equipment (Master’s thesis). Atılım University. https://en.academicresearch.net/development-of-an-electronic-performance-support-system-for- training-people-on-radio-laboratory-equipment/ Askar, A. (2018). Mobile electronic performance support system as a learning and performance solution: A qualitative study examining usage, performance, and attitudes. Turkish Online Journal of Educational Technology, 17(2), 76-88. http://www.tojet.net/articles/v17i2/1728.pdf Barker, P., Schaik, P. V., & Famakinwa, O. (2007). Building electronic performance support systems for first-year university students. Innovations in Education and Teaching International, 44(3), 243-255. https://doi.org/10.1080/14703290701486530 Bollen, L., van der Meij, H., Leemkuil, H., & McKenney, S. (2015). In search of design principles for developing digital learning & performance support for a student design task. Australasian Journal of Educational Technology, 31(5), 500-520. https://doi.org/10.14742/ajet.2546 Brown, L. A. (1996). Designing and developing electronic performance support systems. Digital Press. Carvalho, P. (2018). A model of factors explaining the adoption and use of electronic performance support systems in Portuguese organizations. International Journal of Enterprise Information Systems, 14(1), 111-125. https://doi.org/10.4018/IJEIS.2018010107 Chen, C. H., Hwang, G. J., Yang, T. C, Chen, S. H., & Huang, S. Y. (2009). Analysis of a ubiquitous performance support system for teachers. Innovations in Education and Teaching International, 46(4), 421-433. https://doi.org/10.1080/14703290903301727 Cleary, T. J., & Zimmerman, B. J. (2012). A cyclical self-regulatory account of student engagement: Theoretical foundations and applications. Handbook of research on student engagement. Springer. Costello, E. (2013). Opening up to open source: Looking at how Moodle was adopted in higher education. Open Learning: The Journal of Open, Distance and e-Learning, 28(3), 187-200. https://doi.org/10.1080/02680513.2013.856289 Dias, S. B., Hadjileontiadou, S. J., Diniz, J. A., & Hadjileontiadis, L. J. (2017). Computer-based concept mapping combined with learning management system use: An explorative study under the self-and https://en.academicresearch.net/development-of-an-electronic-performance-support-system-for-training-people-on-radio-laboratory-equipment/ https://en.academicresearch.net/development-of-an-electronic-performance-support-system-for-training-people-on-radio-laboratory-equipment/ http://www.tojet.net/articles/v17i2/1728.pdf https://doi.org/10.1080/14703290701486530 https://doi.org/10.14742/ajet.2546 https://doi.org/10.4018/IJEIS.2018010107 https://doi.org/10.1080/14703290903301727 https://doi.org/10.1080/02680513.2013.856289 Australasian Journal of Educational Technology, 2021, 37(6). 100 collaborative-mode. Computers & Education, 107, 127-146. https://doi.org/10.1016/j.compedu.2017.01.009 Erdoğan, B. (2013). The effect of adaptive learning management system on student's satisfaction, motivation and achievement in online learning (Doctoral dissertation). Ankara University. https://dspace.ankara.edu.tr/xmlui/bitstream/handle/20.500.12575/35370/tez.pdf?sequence=1&isAllo wed=y Erdogmus, F. U. (2015). Design and development of an electronic performance support system for novice instructional designers (Doctoral dissertation). Middle East Technical University. https://etd.lib.metu.edu.tr/upload/12618975/index.pdf Gery, G. (1991). Electronic performance support systems. Weingarten. Henkoğlu, H. Ş., & Mıhçı, P. (2012). The factors affecting the use of educational websites: A scale development study. Ege Eğitim Dergisi, 13(1), 26–39. https://dergipark.org.tr/en/download/article- file/57006 Kalota, F., & Hung, W. C. (2013a). Performance support systems: Design considerations. Proceedings of Current Trends in Information Technology International Conference (pp. 190-193). IEEE. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6749501 Kalota, F., & Hung, W. C. (2013b). Instructional effects of a performance support system designed to guide preservice teachers in developing technology integration strategies. British Journal of Educational Technology, 44(3), 442-452. https://doi.org/10.1111/j.1467-8535.2012.01318.x Kert, S. B., Uz, C., & Gecü, Z. (2014). Effectiveness of an electronic performance support system on computer ethics and ethical decision-making education. Educational Technology & Society, 17(3), 320–331. https://www.jstor.org/stable/pdf/jeductechsoci.17.3.320.pdf Klein, J. D. & Reiser, R. A. (2014). Suggestions for structuring a research article. Journal of Computing in Higher Education, 26(1), 22-38. https://doi.org/10.1007/s12528-013-9075-5 Li, J., & Peng, G. (2016). Development and study of electronic performance support system for capacity development of educational technology of teachers from institutions of higher learning. Revista Iberica de Sistemas e Tecnologias de Informacao, (E7), 366-376. https://search.proquest.com/docview/1838917824/fulltextPDF/3E5CF15B041C4785PQ/1?accountid= 11248 Lizdas, D., Avari, K., Zimmerman, M., & Lampotang, S. (2017, August 27-30). An editor to create integrated tutors for a modular set of mixed reality simulators [Poster paper presentation]. Military Health System Research Symposium, Kissimmee FL. https://simulation.health.ufl.edu/technology- development/augmented-reality-mixed-simulation/ra-sim/abstracts-posters/#prettyPhoto Macfadyen, L., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149-163. https://www.jstor.org/stable/pdf/jeductechsoci.15.3.149.pdf?refreqid=excelsior%3A84c9b1f6fc934e3 3ee9d55caade45560 Martínez-Mediano, C., & Losada, N. R. (2017). Internet-based performance support systems in engineering education. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 12(2), 86-93. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7914728 McKay, J., & Wager, W. (2007). Electronic performance support systems: Visions and viewpoints. In R. A. Reiser, & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology. (3rd ed., pp. 147-155). Merrill/Prentice Hall. Miller, K., Fitzgerald, G., Koury, K., Mitchem, K., & Hollingsead, C. (2007). KidTools: Self- management, problem solving, organizational and planning tools for children and teachers. Intervention in School and Clinic, 43(1), 12-19. https://journals.sagepub.com/doi/pdf/10.1177/10534512070430010201 Mitchell, M. D. (2014). Effectiveness of electronic performance support system and training in a higher education setting (Doctoral dissertation). Capella University. https://search.proquest.com/openview/b8fd77082542d8496fb06aa33e883c4a/1?pq- origsite=gscholar&cbl=18750&diss=y Mitchem, K. J., Fitzgerald, G., & Koury, K. (2016). Electronic performance support system (EPSS) tools to support teachers and students. In M. Khosrow-Pour (Ed.), Special and gifted education: Concepts, methodologies, tools, and applications (pp. 201-222). IGI Global. https://www.igi- global.com/chapter/electronic-performance-support-system-epss-tools-to-support-teachers-and- students/151205 Mitchem, K. J., Fitzgerald, G., Miller, K., & Hollingsead, C. (2013). Using electronic performance support systems to improve academic performance of secondary students with disabilities. Journal of https://doi.org/10.1016/j.compedu.2017.01.009 https://dspace.ankara.edu.tr/xmlui/bitstream/handle/20.500.12575/35370/tez.pdf?sequence=1&isAllowed=y https://dspace.ankara.edu.tr/xmlui/bitstream/handle/20.500.12575/35370/tez.pdf?sequence=1&isAllowed=y https://etd.lib.metu.edu.tr/upload/12618975/index.pdf https://dergipark.org.tr/en/download/article-file/57006 https://dergipark.org.tr/en/download/article-file/57006 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6749501 https://doi.org/10.1111/j.1467-8535.2012.01318.x https://www.jstor.org/stable/pdf/jeductechsoci.17.3.320.pdf https://doi.org/10.1007/s12528-013-9075-5 https://search.proquest.com/docview/1838917824/fulltextPDF/3E5CF15B041C4785PQ/1?accountid=11248 https://search.proquest.com/docview/1838917824/fulltextPDF/3E5CF15B041C4785PQ/1?accountid=11248 https://simulation.health.ufl.edu/technology-development/augmented-reality-mixed-simulation/ra-sim/abstracts-posters/%23prettyPhoto https://simulation.health.ufl.edu/technology-development/augmented-reality-mixed-simulation/ra-sim/abstracts-posters/%23prettyPhoto https://www.jstor.org/stable/pdf/jeductechsoci.15.3.149.pdf?refreqid=excelsior%3A84c9b1f6fc934e33ee9d55caade45560 https://www.jstor.org/stable/pdf/jeductechsoci.15.3.149.pdf?refreqid=excelsior%3A84c9b1f6fc934e33ee9d55caade45560 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7914728 https://journals.sagepub.com/doi/pdf/10.1177/10534512070430010201 https://search.proquest.com/openview/b8fd77082542d8496fb06aa33e883c4a/1?pq-origsite=gscholar&cbl=18750&diss=y https://search.proquest.com/openview/b8fd77082542d8496fb06aa33e883c4a/1?pq-origsite=gscholar&cbl=18750&diss=y https://www.igi-global.com/chapter/electronic-performance-support-system-epss-tools-to-support-teachers-and-students/151205 https://www.igi-global.com/chapter/electronic-performance-support-system-epss-tools-to-support-teachers-and-students/151205 https://www.igi-global.com/chapter/electronic-performance-support-system-epss-tools-to-support-teachers-and-students/151205 Australasian Journal of Educational Technology, 2021, 37(6). 101 Special Education Technology, 28(3), 1-20. https://journals.sagepub.com/doi/pdf/10.1177/016264341302800301 Nguyen, F. (2007). The effect of an electronic performance support system and training as performance interventions (Doctoral dissertation). Arizona State University. https://search.proquest.com/docview/304895017?pq-origsite=gscholar&fromopenview=true Raman, A., Don, Y., Khalid, R., & Rizuan, M. (2014). Usage of learning management system (Moodle) among postgraduate students: UTAUT model. Asian Social Science, 10(14), 186-192. https://doi.org/10.5539/ass.v10n14p186 Reiser, R. A. (2012). What field did you say you were in? In R. A. Reiser, & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (3rd ed., pp. 5-13). Pearson Education. Richey, R. C., Klein, J., & Nelson, W. (2004). Developmental research: Studies of instructional design and development. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (2nd ed., pp. 1099-1130). Lawrence Erlbaum Associates, Inc. Sezer, B., & Yilmaz, R. (2019). Learning management system acceptance scale (LMSAS): A validity and reliability study. Australasian Journal of Educational Technology, 35(3), 15-30. https://doi.org/10.14742/ajet.3959 Ugur-Erdogmus, F., & Cagiltay, K. (2018). Making novice instructional designers expert: Design and development of an electronic performance support system. Innovations in Education and Teaching International, 56(4), 1-11. https://doi.org/10.1080/14703297.2018.1453853 Corresponding author: Baris Sezer, barissezer@hacettepe.edu.tr Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. Please cite as: Sezer, B. (2021). Developing and investigating an EPSS for academic performance. Australasian Journal of Educational Technology, 37(6), 88-101. https://doi.org/10.14742/ajet.6121 https://journals.sagepub.com/doi/pdf/10.1177/016264341302800301 https://search.proquest.com/docview/304895017?pq-origsite=gscholar&fromopenview=true https://doi.org/10.5539/ass.v10n14p186 https://doi.org/10.14742/ajet.3959 https://doi.org/10.1080/14703297.2018.1453853 https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6121