P-ISSN: 1412-1212 E-ISSN: 2541-2388 75*Corresponding Author The Winners, 21(2), September 2020, 75-83 DOI: 10.21512/tw.v21i2.6505 Performance Measurement of Academic Information Systems using Performance Prism and ISO/IEC 25010 Liulliyah1*; Apol Pribadi Subriadi2 1,2Information System Department, Institut Teknologi Sepuluh Nopember Jl. Teknik Kimia, Keputih, Kec. Sukolilo, Surabaya 60111, Indonesia 1ailyulil@gmail.com; 2apolpribadi@gmail.com Received: 16th June 2020/ Revised: 14th July 2020/ Accepted: 27th July 2020 How to Cite: Liulliyah & Subriadi, A. P. (2020). Performance measurement of Academic Information Systems using performance prism and ISO/IEC 25010. The Winners, 21(2), 75-83. https://doi.org/10.21512/tw.v21i2.6505 Abstract - The aim of the research was to measure system performance information in terms of system quality using the ISO/IEC 25010 model and involve stakeholders using the Performance Prism (PP) model. The research applied an action research approach where researchers developed performance measurement models and solved existing problems. The research produced 41 performance indicators, which were used to measure academic information system. The results in general academic information systems have a performance value of 5,522. Keywords: Academic Information Systems, performance measurement, performance prism, ISO/ IEC 25010 I. INTRODUCTION Organizations currently regard information systems (IS) as investment capital and not operating costs (Farbey, Land & Targett, 1993). The high potential of the role of IS in improving business performance, causes companies to increase the IS budget significantly enough to develop and maintain IS (Seddon, Graeser & Willcocks, 2000). IS is not only used to replace manual processing, but is used for more important things namely strategic objectives, such as decision making and gaining competitive advantage (Laudon & Laudon, 1996). On average 200 companies spend 20-40% of their operating budgets on IT to stay competitive (May, 2001). Organizations make large investments in IS by expecting a positive impact on the organization (Sedera & Tan, 2005). However, in practice, IS investments are rarely systematically evaluated post-implementation. When post-implementation evaluations occur, the processes and steps are often exceptional and lack credibility or comparability (Thatcher & Oliver, 2001). The success of an organization in obtaining benefits from IS cannot be separated from the success of the IS function in carrying out its duties. It is therefore necessary for organizations to regularly assess the performance of their IS functions. This leads to the need for knowledge about how to evaluate the performance of IS functions comprehensively (Chang & King, 2005). In addition to ensuring that organizations get value from what they spend, IS performance must be evaluated (Sulaiman, 1996). Performance evaluation is carried out to assess the extent to which information system performance increases effectiveness and efficiency in achieving institutional strategic objectives so that steps can be taken to improve or innovate the latest technology and help maximize investment in information technology (Agani, Munadi & Subianto, 2018). Moreover, IS performance must be evaluated to control quality and for audit purposes (Sulaiman, 1996). However, evaluating IS is a complicated process, the main problem is how to identify performance measures that must be used to evaluate performance (Symons, 1990). Performance measurement itself can be interpreted as a set of matrices that are used to calculate efficiency and effectiveness in a series of actions (Neely, Gregory & Platts, 1995). Stakeholders’ views on the ability of information system functions can influence the willingness of these stakeholders to propose information technology innovation (Ravichandran & Lertwongsatien, 2005; Zhu, 2004). Although involving stakeholders is complex and time-consuming, the benefits are more significant because by involving stakeholders the risk of losing specific information is smaller (Delnoij et al., 2010). Stakeholder analysis must be used as one important 76 The Winners, Vol. 21 No. 2 September 2020, 75-83 evaluation (Farbey, Land & Targett, 1999), and understand stakeholder strategies that are useful and can influence the success of information systems (Vaidya, Myers & Gardner, 2011). One method for measuring performance involving stakeholders is the Performance Prism method (PP). What distinguishes PP from other frameworks is that PP has unique and important features (Neely, Adams & Crowe, 2001a). PP has five aspects, namely for the top and bottom is the satisfaction of stakeholders and stakeholder contributions. Whereas the next three sides are strategy, process, and capability (Neely & Adams, 2000). However, this PP is used by several studies to measure organizational performance (Smulowitz, 2015; Gardoni, 2017; Yu & Zhu, 2011), even though one way to measure the benefits of the system is to assess the quality of the system (Al- Hudhaif & Arabia, 2010). Apart from the perspective of users/stakeholders, IS also needs to be measured in terms of product quality and service quality, especially on the “capability” of the IS function itself (Chang & King, 2005). This is done to obtain the right size that can later influence the decision making process in terms of developing IS services considering the IS in each organization has different levels (Davies et al., 1999). Several models can be used to measure the quality of IS including the McCall quality model, the Boehm quality model, the Dromey quality model, and ISO/IEC 9126 (Sulistiani et al., 2018). Whereas ISO/ IEC 25010 is an International standard in software testing. The ISO/IEC 25010 standard was developed to replace ISO 9126 based on the development of ICT (Information and Communication Technology) (ISO/ IEC 25010, 2011). With ISO/IEC 25010 allows more intensive analysis and can obtain critical characteristics (Darwish & Shehab, 2017). PP is a theory developed by Cranfield University, then Adams and Neely (2000) introduces it as a method of measuring company performance. The strength of the framework is that it involves all stakeholders from the organization. In principle, this method has two directions in the process which is the reciprocal relationship of each stakeholder, namely by considering the needs, interests and desires of all stakeholders, and identifying the contribution of stakeholders to the organization. PP has been implemented in several companies including DHL UK, The London Youth and The House of Fraser. The result is that some of these companies can have clear performance measurement parameters so that the company can have remarkable development. Jun (2011) uses PP to measure ERP software. Jun (2011) states PP is a beneficial valuation model in providing financial management that is easy to understand. Besides PP is not only oriented towards interests since it also measures the contribution made by stakeholders to the organization. Yu and Zhu (2011) use PP to evaluate informatization of chemical companies and using fuzzy mathematical assessment methods for quantity of qualitative indices and stating that the evaluation system using PP has good practical operability because it uses all stakeholders and their four facets namely stakeholder needs, organizational strategy, business processes, and stakeholder contributions as a consideration. Laili (2012) uses PP to measure the performance of the e-KTP program, the results are that there are four key stakeholders who have 34 Key Performance Indicators (KPI). 20 KPIs fall into the green category, 12 KPIs fall into the yellow category, and 2 KPIs fall into the red category. While the total value of e-KTP performance is 6,441. Using PP in tertiary institutions is very good, according to Smulowitz (2015) because universities have different system characteristics. PP can evaluate all the ‘wants and needs’ of stakeholders who might be ignored. Whereas the ‘wants and needs’ of stakeholders can be the key to success of higher education in planning for sustainability. The application of PP in small and medium companies shows significant results (Severgnini, Galdamez & Camacho, 2019). Small and medium companies have limitations and difficulties in observing the overall strategy besides the contribution of stakeholders to the existing business is also low. PP is practiced in the social service sector and (Estrada, Sousa & Lopes, 2018) states that PP is a motivating tool for organizations to make internal reflections of their processes. Through this process, organizations can identify the importance of achieving goals so they can determine indicators to better control their performance. Youngbantao and Rompho (2015) apply PP to several companies by involving corporate organizational culture, namely a culture of flexibility and stability. The result shows no significant difference in the application of PP in the two types of organizational culture, but in its application, the five perspectives that have the highest level of importance are stakeholder satisfaction. PP has also been developed by Wasitarini and Sembiring (2017) by modifying it using the Objective Matrix (OMAX) and 3600 feedback in the form of a Quantitative Models for Performance Measurement System (QMPMS) to measure the performance of the Electronic Library (E-Library), the design modification model successfully identified Key Performance Indicators (KPI). ISO/IEC 25010 is an International standard in software measurement. ISO/IEC 25010 was created in order to revise the previous software quality model and to overcome weaknesses in the previous model, namely ISO/IEC 9126 (ISO, 2011). ISO/IEC 9126 is based on the McCall and Boehm quality models (ISO/ IEC IS 9126-1, 2001). ISO/IEC 25010 enables more intensive analysis to obtain critical characteristics (Darwish & Shehab, 2017). Kara, Lamouchi and Ramdane-Cherif (2017) use ISO 25010 and the concept of fuzzy logic to evaluate the Ambient Assisted Living System. The evaluation results can show the necessary quality models and produce quality indicators that can help 77Performance Measurement of Academic ..... (Liulliyah; Apol Pribadi Subriadi) users make the right decisions at the same time. Huda (2012) integrates Webqual 4.0 with ISO/ IEC 25010. Measurement of academic and financial information systems produces more complete indicators so that the assessment of software quality is better. Lesmideyarti, Rochimah, and Yuhana (2017) use ISO/IEC 25010 to measure the performance of academic information systems and compare them with academic information system measurements using ISO/IEC 9162. The results of measurements using ISO/IEC 25010 are more accurate because the characteristics of ISO/IEC 25010 are more complete. Haslinda et al. (2015) assess the quality of e-book applications in facilitating learning in formal education based on ISO 25010 standards. The survey was conducted in 37 elementary schools involving 200 teachers. The results show that e-book can be used, reliably, functionally, and efficiently. In addition, e-books occurs to fulfill the purposes and objectives of the e-book procurement itself. Karnouskos et al. (2018) use the ISO/IEC 25010 software product quality model in industrial automation systems by conducting a brief survey of industry automation experts, because there is currently no set of general criteria for assessing and identifying practices that are most suitable for certain industrial uses. From this research, it can be seen what quality is very relevant. ISO/IEC 25010 is also used as a measurement of academic information systems by Puspaningrum, Rochimah and Akbar (2017), but they add the Goal Question Metrics (GQM) in applying the assessment questionnaire so that the quality value increased and according to the conditions and needs of higher education today. In some of the research mentioned earlier, many researchers have measured information system performance. However, measuring the performance of information systems requires a model that can measure the quality of the system and involve the views of stakeholders. The objective of the research is to create a model that can measure the quality of information systems and involve stakeholders. For this reason, researchers use the performance prism model to facilitate stakeholders, and ISO/IEC to measure the quality of the system. The model will be implemented in an academic information system (SIAKAD) at the Poltekkes Kemenkes Surabaya. The results of the research can be used by organizations to develop their SIAKAD because the performance measurement model can show which parts of SIAKAD are performing well or are problematic for finding solutions. II. METHODS The research uses an action research approach with the aim of solving practical problems while developing knowledge (Sarosa, 2017). Coats (2005) mentions that action research is about ‘action’ and ‘research’ and the relationship between the two. It is possible to take action without research or to conduct research without taking action, but what is unique is to combine the two, which is to conduct research and test action. In the research, there are three stages in accordance with the statement (Coghlan, 2005) in his book on action research. The stages of the research are planning, taking action and evaluation. Planning, in the research, is done by creating a conceptual model of research that can produce performance measurement models. The next stage is taking action, where the performance measurement model that has been made is implemented on the selected object, namely the academic information system of the Poltekkes Kemenkes Surabaya. In the final stage, which is evaluation, the results of the implementation of the performance measurement model are analyzed in such a way that recommendations can be given for the development of the academic information system and the emergence of new performance measurement models that have passed the stages of implementation and validation. III. RESULTS AND DISCUSSIONS Based on literature studies and cases examined in the field, in general, the construct of the research model is shown in Figure 1. Figure 1 Conceptual Model Figure 1 is a conceptual model in measuring the performance of SIAKAD using PP and ISO/IEC 25010. The PP method here adjusts its basic form which consists of five sides, namely stakeholder satisfaction, stakeholder contribution, strategies, process, and capability. Stakeholder satisfaction and satisfaction contribution have a reciprocal relationship by considering the needs and desires of all stakeholders, as well as identifying the contribution of stakeholders to the organization. The reciprocal process will be supported by strategies, processes, and capability so that the five sides will be able to interact with each other. PP will produce performance indicators by conducting interviews with stakeholders involved 78 The Winners, Vol. 21 No. 2 September 2020, 75-83 regarding satisfaction, contribution, strategies, process, and capability using the format of questions in Table 1. The resulting performance indicators are compared with the performance indicators of ISO/IEC 25010. Performance indicators of comparative results will be given weighting and performance measurement analysis. The analysis will produce recommendations for measured performance. Table 2 is a performance indicator resulting from the comparative process of PP performance indicators and ISO/IEC 25010 performance indicators. To measure the performance of each indicator, a Likert questionnaire survey of 48 respondents has been conducted using purposive sampling. The performance indicator of comparative results is weighed using AHP weighting carried out by 15 respondents with the result of weighting being the result of a combination of all respondents listed in Table 3. Table 1 Format of PP Interview Questions (Neely et al., 2000) Facet Question Satisfaction What do you want and need from SIAKAD? Strategy According to you, what strategies can be done to fulfill those wants and needs? Process What processes can be implemented to realize the strategy? Capabilities What abilities must SIAKAD have for this process to be carried out? Contribution What kind of contributions can you make to SIAKAD in supporting the process? Table 2 Performance Indicators Number Performance Indicator Code 1 The ability of SIAKAD to cover all tasks and objectives as specified IK-01 2 The ability of SIAKAD to provide correct results with the required level of precision. IK-02 3 The ability of SIAKAD in facilitating the completion of tasks and goals that have been determined IK-03 4 The ability of SIAKAD to provide reaction and processing time when performing functions IK-04 5 The level of use of resources in SIAKAD in carrying out its functions IK-05 6 The maximum limit or SIAKAD parameter meets the requirements IK-06 7 SIAKAD can perform the necessary functions efficiently while sharing the same environment and resources with other products, without harming other products. IK-07 8 The ability of SIAKAD to exchange information and use information that has been exchanged IK-08 9 The ability of SIAKAD is analyzed by the user whether the system can meet the needs IK-09 10 SIAKAD is easy to understand and learn by users IK-10 11 SIAKAD is easy to operate and access IK-11 12 SIAKAD protects users from making mistakes. IK-12 13 SIAKAD's appearance attracts the user's attention IK-13 14 The ability of SIAKAD to meet the needs of users who have limitations IK-14 15 The ability of SIAKAD to bear, handle or cover up failures and errors that occur IK-15 16 SIAKAD operates and can be accessed when needed for use IK-16 17 SIAKAD's ability to maintain performance against software errors or violations by users IK-17 18 The ability of SIAKAD to restore data manually immediately if something goes wrong or fails IK-18 19 The ability of SIAKAD to ensure that data can only be accessed by those authorized to have access. IK-19 20 SIAKAD's ability to prevent unauthorized access to, or modification of, computer programs or data IK-20 21 The ability of SIAKAD to prove the actions that have occurred IK-21 22 The ability of SIAKAD to track actions that occur IK-22 23 The ability of SIAKAD to demonstrate the subject's identity or resources as claimed IK-23 24 Changes to one component of SIAKAD have minimal impact on other components. IK-24 25 An asset can be used in more than one system, or in building other assets IK-25 26 The ability of SIAKAD to diagnose the cause of failure and identify failure IK-26 79Performance Measurement of Academic ..... (Liulliyah; Apol Pribadi Subriadi) The results of the questionnaire calculation and AHP weighting obtained are included in the scoring objective matrix (OMAX) table in Appendix A, with the questionnaire calculation value as the performance value of each performance indicator while the AHP weighting as weight. The formula for calculating intervals on OMAX (Riggs, 1987) is: ∆X L-H = YH-YL (1) XH-XL Notes: ΔX L-H = Number interval between High and Low levels X H = High level X L = Low level Y H = Number at the high level Y L = Number at the low level Based on the Likert scale questionnaire results, the performance indicator has an achievement target of 5. This value is used as the highest scale value of the objective matrix, which is 10. While the lowest possible value of this performance indicator is 1. Therefore by using the OMAX formula, the interval calculation between scales becomes: Interval scale = 5-1 10-0 = 0,4 From the OMAX scoring table using the traffic light system in Appendix A, it can be analyzed that only one performance indicator reaches green, namely, the IK-41 performance indicator, while the performance indicators that are at a low level or red are two indicators namely IK-14 and IK-40 performance indicators, while most indicators are still yellow, i.e., Number Performance Indicator Code 27 The ability of SIAKAD to be modified without reducing the quality of existing products IK-27 28 The SIAKAD testing process IK-28 29 The ability of SIAKAD to operate in diverse operating environments without applying other actions or rules IK-29 30 The ability of SIAKAD to be installed/removed in certain environments IK-30 31 The ability of SIAKAD to be transferred to other software that has been built or determined IK-31 32 Ease of system maintenance IK-32 33 The ability of the system to adjust the existing budget IK-33 34 The ability of the system to adapt the existing architecture IK-34 35 HR ability to carry out academic activities on schedule IK-35 36 The ability of SIAKAD to be transferred to other software that has been built or determined IK-36 37 Ease of system maintenance IK-37 38 The ability of the system to adjust the existing budget IK-38 39 The ability of the system to adapt the existing architecture IK-39 40 The ability of Human Resources to carry out academic activities on schedule IK-40 41 The ability of SIAKAD to be transferred to other software that has been built or determined IK-41 Table 3 Weight of Performance Indicators Performance Indicators IK-01 IK-02 IK-03 IK-04 IK-05 IK-06 IK-07 IK-08 IK-09 IK-10 IK-11 IK-12 Weight 0,015 0,029 0,016 0,020 0,018 0,017 0,019 0,019 0,019 0,040 0,044 0,027 Performance Indicators IK-13 IK-14 IK-15 IK-16 IK-17 IK-18 IK-19 IK-20 IK-21 IK-22 IK-23 IK-24 Weight 0,015 0,018 0,023 0,021 0,022 0,025 0,041 0,045 0,028 0,025 0,018 0,019 Performance Indicators IK-25 IK-26 IK-27 IK-28 IK-29 IK-30 IK-31 IK-32 IK-33 IK-34 IK-35 IK-36 Weight 0,015 0,024 0,024 0,018 0,014 0,020 0,018 0,024 0,019 0,024 0,023 0,028 Performance Indicators IK-37 IK-38 IK-39 IK-40 IK-41 Weight 0,031 0,033 0,025 0,034 0,044 Table 2 Performance Indicators (Continued) 80 The Winners, Vol. 21 No. 2 September 2020, 75-83 38 performance indicators (Appendix A). For a performance indicator that gets a green traffic light indicator, it can be said that the performance indicator has reached the target or at least has been close to it, so it does not need much evaluation and improvement regarding the problem of the performance indicator while the yellow performance indicator is the middle value of the indicator. Performance indicators that have yellow indicators need to be identified and further evaluated related to the performance issues of these performance indicators. Stakeholders must analyze what makes the value of the indicator decrease and what can make the value of the indicator increase. Lastly, red indicators shows the performance indicators that are at a low score level so that they need immediate improvement. The total performance index value of SIAKAD based on appendix A table is 5,522. If implemented using a traffic light system, then the value is still included in the yellow value category, so it can be concluded that the overall performance of SIAKAD needs evaluation and improvement in it. After implementing the initial model for measuring the performance of the SIAKAD, the position of stakeholder contributions has changed to align with strategies, processes, and capabilities as shown in Figure 2. Figure 2 Performance Measurement Model Stakeholder satisfaction becomes important to identify system performance measurements as it will be supported by stakeholder contribution, strategies, process, and capability. This was also conveyed by Neely and Adams (2000) that the needs and desires of the stakeholders must be considered first, so new strategies can be formulated. For performance indicators with a low score in red, there are several recommendations for improvement where institutions must quickly take action to improve the performance of these indicators. For indicator IK-14 using guidelines on the Web Content Accessibility Guidelines (WCAG) 2.0 published by the World Wide Web Consortium (W3C) can be implemented at SIAKAD. Some of these guidelines include: a) providing alternative text for any non-text content so that it can be transformed into other forms that people need, such as large print, braille, speech, symbols, or simpler language; b) provides alternatives for time-based media; c) creating content that can be presented in different ways (for example a simpler layout) without losing information or structure; d) enabling users to easily see and hear content including separating the foreground from the background; e) making all functionality available from the keyboard; f) provide users enough time to read and use content; g) not designing content in a way that is known to cause seizures; h) providing ways to help users navigate, find content, and determine where they are; i) making text content easy to read and understand; j) enabling webpages to appear and operate in predictable ways; k) helping users avoid and correct errors; l) maximizing compatibility with current and future user agents, including assistive technology. For IK-40 indicators by providing training by the required fields of existing IT human resources, if it is deemed lacking, recruitment of IT human resources should be carried out by the requirements. IV. CONCLUSIONS By using the PP and ISO/IEC 25010 methods, 41 performance indicators can be obtained which can be used to measure SIAKAD performance. This performance indicator has been implemented in SIAKAD Poltekkes Ministry of Health Surabaya. The result shows that one performance indicator with good or achieved performance, 38 performance indicators with moderate/unperformed performance and in the recommended warning conditions to improve performance, and two performance indicators with poor performance scores that require improvement as both indicators are at a low score. While the total value of the SIAKAD performance index is 5,522. These results can be used by organizations as a description of the performance of higher education if the achievement targets have not been achieved to be used as evaluation material. Moreover, it may show the organization the aspects of the information system that need improvement. Hence the organization can make the results as a reference to develop its SIAKAD performance. The measurement model produced in the research has only been implemented in academic information systems. Further research can aim to implement it in other information systems and with different assessment methods. REFERENCES Adams, C., & Neely, A. (2000). The performance prism to boost M&A success. Measuring Business Excellence, 4(3), 19-23. https://doi. org/10.1108/13683040010377818. Agani, M. A., Munadi, R., & Subianto, M. (2018). Evaluasi 81Performance Measurement of Academic ..... (Liulliyah; Apol Pribadi Subriadi) kinerja sistem informasi akademik menggunakan IT balanced scorecard pada universitas Serambi Mekkah Banda Aceh. Jurnal Informatika Upgris, 4(1). https://doi.org/10.26877/JIU.V4I1.2071. Al-Hudhaif, S. A., & Arabia, S. (2010). Measuring quality of information system services in manufacturing organizations in Riyadh. JKAU: Econ. & Adm, 24(1), 151-171. https://doi.org/10.4197/Eco. Chang, J. C-J & King W. R. (2005). Measuring the performance of information system: A functional scorecard. Journal of Mangement Information Systems, 22(1), 85-115. https://doi.org/10.1080/074 21222.2003.11045833. Coats, M. (2005). Action research a guide for associate lecturers. UK: Walton Hall Milton Keynes. h t t p : / / r e p o s i t o r i o . m i n e d u . g o b . p e / b i t s t r e a m / handle/123456789/3590/Action%20Research%20 A%20Guide%20for%20Associate%20Lecturers. pdf?sequence=1&isAllowed=y. Coghlan, D. (2005). Doing action research in your own organization. SAGE Publications. Darwish, M., & Shehab, E. (2017). Framework for engineering design systems architectures evaluation and selection: Case study. Procedia CIRP, 60, 128- 132. https://doi.org/10.1016/j.procir.2017.01.058. Davies, B., Baron, S., Gear, T. & Read, M. (1999). Measuring and managing service quality. Marketing Intelligence & Planning. 17(1), 33-40. https://doi. org/10.1108/02634509910253795. Delnoij, D. M. J., Rademakers, J. J. D. J. M., & Groenewegen, P. P. (2010). The Dutch consumer quality index: An example of stakeholder involvement in indicator development. BMC Health Services Research, 10. https://doi.org/10.1186/1472-6963-10-88. Estrada, R., Sousa, S., & Lopes, I. (2018). Performance assessment system development based on performance prism in social services. IEEE International Conference on Industrial Engineering and Engineering Management, 2017-December, 855- 859. https://doi.org/10.1109/IEEM.2017.8290013. Farbey, B., Land, F., & Targett, D. (1999). Moving IS evaluation forward: Learning themes and researchissues. Journal of Strategic Information Systems, 8(2), 189-207. https://doi.org/10.1016/ S0963-8687(99)00021-9. Farbey, B., Land, F., & Targett, D. (1993). How to assess your IT investment. UK: Butterworth-Heinemann Ltd. Gardoni, F. L. (2017). Double performance prism: Innovation performance measurement systems for manufacturing SMEs. International Journal of Innovative Research in Advanced Engineering, 4(3), 113-128. Haslinda, H., Fahmy, F., Sukinah, S., Roslina, R., Fariha, F., Suhana, S., Azliza, A., & Shiratuddin, N. (2015). Evaluation of e-Book applications using ISO 25010. 2nd International Symposium on Technology Management and Emerging Technologies, ISTMET 2015 - Proceeding, December, 114-118. https://doi. org/10.1109/ISTMET.2015.7359012. Huda, D. N. (2012). Peningkatan kualitas sistem informasi akademik dan keuangan berdasarkan Webqual 4.0 dan ISO/IEC 25010: Studi kasus Sekolah Tinggi Teknologi Indonesia Tanjungpinang. Jurnal Bangkit Indonesia, 6(2), 37-39. https://journal.sttindonesia. ac.id/index.php/bangkitindonesia/article/view/21. ISO/IEC 25010. (2011). ISO/IEC. Systems and software engineering-Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. BSI Standards Publication. https://www.iso.org/standard/35733. html ISO/IEC IS 9126-1. (2001). Software Engineering - Product Quality – Part 1: Quality Model. International organization for standarization. https://www.iso.org/ standard/22749.html ISO. (2011). Systems and software engineering Systems and software Quality Requirements and Evaluation. 6(1.3.2011), 1-44. https://www.iso.org/obp/ ui/#iso:std:iso-iec:25010:ed-1:v1:en Jun, S. (2011). BMEI 2011 - Proceedings 2011 International Conference on Business Management and Electronic Information. BMEI 2011 - Proceedings 2011 International Conference on Business Management and Electronic Information, 1. Kara, M., Lamouchi, O., & Ramdane-Cherif, A. (2017). A quality model for the evaluation AAL systems. Procedia Computer Science, 113, 392-399. https:// doi.org/10.1016/j.procs.2017.08.354. Karnouskos, S., Sinha, R., Leitão, P., Ribeiro, L., & Strasser, T. I. (2018). The applicability of ISO/IEC 25023 measures to the integration of agents and automation systems. IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society, January 2019, 2927–2934. https://doi.org/10.1109/ IECON.2018.8592777. Laili, V. B. (2012). Perancangan sistem pengukuran kinerja progam e-KTP sebagai media analisis investasi IT dengan menggunakan metode performance prism. Institut Teknologi Sepuluh Nopember. http://digilib. its.ac.id/ITS-paper-52021120000168/19415. Laudon, K. & Laudon, J. (1996). MILS. Organization and Technology. New Jersey: Prentice Hall. Lesmideyarti, D., Rochimah, S., & Yuhana, U. L. (2017). Penyusunan dan pengujian metrik operabilitas untuk sistem informasi akademik berdasarkan ISO 25010. Inspiration: Jurnal Teknologi Informasi dan Komunikasi, 7(2), 92-100. https://doi.org/10.35585/ inspir.v7i2.2442. May, T. (2001, March 12th). Annual IT Budget? Kill It: CIOs need Net-style flexibility for enterprise projects. Computer World. https://www.computerworld.com/ article/2590963/annual-it-budget--kill-it.html Neely, A. & Adams, C. (2000). Perspectives on performance: The performance prism. Focus Magazine for the Performance Management Professional, 4. https:// www.som.cranfield.ac.uk/som/dinamic-content/ research/cbp/prismarticle.pdf. Neely, A., Adams, C., & Crowe, P. (2001a). The performance prism in practice. Measuring Business Excellence, 5(2), 6-13. https://doi. org/10.1108/13683040110385142. 82 The Winners, Vol. 21 No. 2 September 2020, 75-83 Neely, A., Adams, C., & Crowe, P. (2001b). The performance prism in practice. Measuring Business Excellence, 5(2), 6-13. https://doi. org/10.1108/13683040110385142. Neely, A. D., Gregory, M., & Platts, K. (1995). Performance measurement system design: a literature review and research agenda. International Journal of Operations & Production Management, 15(4), 80-116. https:// doi.org/10.1108/01443579510083622. Puspaningrum, A. S., Rochimah, S., & Akbar, R. J. (2017). Functional suitability measurement using goal-oriented approach based on ISO/IEC 25010 for Academics Information System. Journal of Information Systems Engineering and Business Intelligence, 3(2), 68. https://doi.org/10.20473/ jisebi.3.2.68-74. Ravichandran, T., & Lertwongsatien, C. (2005). Effect of information systems resources and capabilities on firm performance: A resource-based perspective. Journal of Management Information Systems, 21(4), 237-276. https://doi.org/10.1080/07421222.2005.11 045820. Riggs, J. L. (1987). Production system: Planning, analysis & control. New York: Jhon Wiley & Sons. Sarosa, S. (2017). Penelitian Kualitatif Dasar-Dasar, Edisi 2 (2nd Ed.). Indeks Jakarta. Seddon, P. B., Graeser, V., & Willcocks, L. (2000). Measuring IS effectiveness: Senior IT management perspectives. Oxford Institute of Information Management. Sedera, D., & Tan, F. T. C. (2005). User Satisfaction: An overarching measure of Enterprise System success. 9th Pacific Asia Conference on Information Systems: I.T. and Value Creation, PACIS 2005, 2, 963-976. Severgnini, E., Galdamez, E. V. C., & Camacho, R. R. (2019). Applicability of Performance Prism in SMEs: a multiple case study. Gestão & Produção, 26(4). https://doi.org/10.1590/0104-530x3014-19. Smulowitz, S. (2015). Evidence for the performance prism in higher education. Measuring Business Excellence, 19(1), 70-80. http://dx.doi.org/10.1108/MBE-11- 2014-0043. Sulaiman, A. (1996). A review of information systems performance evaluation measures. Journal of Business and Social Sciences, 11(1), 1-16. Sulistiani, I., Syarif, S., Yusran & Dewiani. (2018). Quality Models Engineering for Evaluation of Academic Information System Quality Instrument (AISQI). Proceedings - 2018 International Conference on Applied Science and Technology, ICAST 2018, 30- 34. https://doi.org/10.1109/iCAST1.2018.8751525. Symons V. J. (1990). A review of IS evaluation: Content, context and process. European Journal of IS, 1(3), 205.212. Thatcher, M. E., & Oliver, J. R. (2001). The impact of technology investments on a firm’s production efficiency, product quality, and productivity. Journal of Management Information Systems, 18(2), 17-45. Vaidya R., Myers M.D., & Gardner L. (2011). The design – reality gap: The impact of stakeholder strategies on IS implementation in developing countries. In Nüttgens M., Gadatsch A., Kautz K., Schirmer I., Blinn N. (eds) Governance and sustainability in information systems. Managing the Transfer and Diffusion of IT. TDIT 2011. IFIP advances in information and communication technology, 366. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642- 24148-2_8. Wasitarini, D. E., & Sembiring, J. (2017). Performance measurement framework of E-Library using modified quantitative models for performance measurement system (QMPMS) method for ICT infrastructure. 2016 International Conference on Information Technology Systems and Innovation, ICITSI 2016 - Proceedings. https://doi.org/10.1109/ ICITSI.2016.7858192. Youngbantao, U., & Rompho, N. (2015). The uses of measures in Performance Prism in different organizational cultures. Journal of Accounting & Finance, 15(6), 122-128. https://www.researchgate. n e t / p u b l i c a t i o n / 2 8 5 8 2 6 2 3 0 _ T h e _ U s e s _ o f _ Measures_in_Performance_Prism_in_Different_ Organizational_Cultures. Yu, B. & Zhu, B. (2011). Design of performance evaluation system for the informatization of chemical enterprise based on Performance Prism. 2011 IEEE 3rd International Conference on Communication Software and Networks, Xi’an, pp. 394-396. https:// doi.org/10.1109/ICCSN.2011.6013857. Zhu, K. (2004). The complimentary of Information Technology infrastructrure and E-Commerce Capability: A resource-based assessment of their business value. Journal of Management Information Systems, 21(4), 167-202. https://doi.org/10.1080/07 421222.2004.11045794. 83Performance Measurement of Academic ..... (Liulliyah; Apol Pribadi Subriadi) Appendix A Performance Indicator Performance Level Score Skor Bobot Value Total 10 9 8 7 6 5 4 3 2 1 0 IK-01 3,44 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,015 0,090 5,552 IK-02 3,59 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,029 0,174 IK-03 3,5208 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,016 0,096 IK-04 3,5521 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,020 0,120 IK-05 3,5625 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,018 0,108 IK-06 3,5521 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,017 0,102 IK-07 3,7292 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,019 0,114 IK-08 3,5417 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,019 0,114 IK-09 3,4167 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,019 0,114 IK-10 3,3281 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,040 0,200 IK-11 3,4583 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,044 0,264 IK-12 3,1528 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,027 0,135 IK-13 3,2917 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,015 0,075 IK-14 2,5729 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 3 0,018 0,054 IK-15 2,9583 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 4 0,023 0,092 IK-16 3,3542 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,021 0,105 IK-17 3,3125 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,022 0,110 IK-18 3,2396 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,025 0,125 IK-19 3,7708 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 7 0,041 0,287 IK-20 3,5625 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,045 0,270 IK-21 2,875 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 4 0,028 0,112 IK-22 3,6042 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,025 0,150 IK-23 3,9792 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 7 0,018 0,126 IK-24 3,2083 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,019 0,095 IK-25 3,3229 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,015 0,075 IK-26 3,6146 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,024 0,144 IK-27 3,4583 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,024 0,144 IK-28 3,2396 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,018 0,090 IK-29 3,2847 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,014 0,070 IK-30 3,1771 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,020 0,100 IK-31 3,4583 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,018 0,108 IK-32 3,2292 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,024 0,120 IK-33 3,4375 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,019 0,114 IK-34 3,2917 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,024 0,120 IK-35 3,4792 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,023 0,138 IK-36 3,375 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,028 0,140 IK-37 2,9167 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 4 0,031 0,124 IK-38 3,3854 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 5 0,033 0,165 IK-39 3,4792 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 6 0,025 0,150 IK-40 2,4375 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 4 0,034 0,136 IK-41 4,4167 5 4,6 4,2 3,8 3,4 3 2,6 2,2 1,8 1,4 1 8 0,044 0,352