Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4, 926-934 2024 Publisher: Learning Gate DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate © 2024 by the authors; licensee Learning Gate * Correspondence: ditdit.utama@binus.edu.ac.id Enhancing IT maturity in the DevOps focus area: a comprehensive assessment with COBIT 2019 Febby Febrianti1, Ditdit Nugeraha Utama2* 1,2Binus Graduate Program, Master of Computer Science, Bina Nusantara University Jakarta, Indonesia 11480; febby.febrianti@binus.ac.id, ditdit.utama@binus.edu.ac.id; (F.F.) ditdit.utama@binus.edu.ac.id (D.N.U.) Abstract: The research will investigate the following research inquiries: by utilizing the COBIT 2019 design toolkit, what is the level of DevOps maturity within the company according to COBIT 2019, and what recommendations are required to enhance the maturity level in DevOps. The researchers conducted a collaborative workshop with the critical management team to obtain reliable and accurate information. The workshop engaged participants from the development and operations teams to gather and analyze data about current DevOps practices. Based on the data, it is known that out of the 26 processes within the DevOps focus area, they have been measured and resulted in a maturity level of 3.15. The value of 3.15 indicates that the DevOps maturity level falls under the ''Defined'' category, meaning that the processes within the DevOps focus area have been implemented consistently. The case study research has reached its outcomes, and the researcher can conclude that the study has successfully achieved its objectives and benefits. This research has conducted an assessment of the maturity level for the DevOps focus area using the Process Assessment Model (PAM) as a measurement tool. Keywords: COBIT 2019, DevOps, Focus area, Maturity level, Process Assessment model (PAM), 1. Introduction Implementing DevOps requires cultural, process, and tooling changes that support collaboration between development and operations teams. However, the success of DevOps implementation can be influenced by various factors, such as organizational complexity, business needs, and risk management (Lestari et al., 2022). In this context, the COBIT 2019 framework can be used to assist organizations in managing and integrating DevOps practices. COBIT 2019 is a comprehensive framework for IT governance. In the context of DevOps, the design factors in COBIT 2019 can assist organizations in identifying relevant domains and objectives related to DevOps practices (Ramaj et al., 2022). Design factors, such as organizational culture, structure, and technological needs, can help organizations build a governance system that aligns with their specific requirements and goals. In the implementation of DevOps, organizations need to consider the following design factors: • Organizational Culture: Adopting a culture of collaboration, transparency, and shared responsibility between development and operations teams(Zarour et al., 2019). • Enhancing Organizational Structure: Establishing cohesive cross-functional teams to eradicate obstacles and foster collaborative synergy. • Processes and Tools: Adopting tools and processes that support automation, continuous testing, and rapid delivery. While COBIT 2019 can provide valuable guidance in implementing DevOps (Lestari et al., 2022), some challenges may arise, including: 927 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate • Cultural Change: Transforming existing organizational culture and adopting a collaborative DevOps culture. • Alignment with Existing DevOps Practices: Integrating existing DevOps practices with the guidance and objectives outlined in COBIT 2019. • Change Management: Managing the changes in organizational structure, processes, and tools with full support from senior management. The Author aims to analyze the organization's DevOps maturity level by employing the COBIT 2019 design toolkit. Drawing upon the findings, suggestions will be presented to elevate the level of maturity and rectify any identified deficiencies. These recommendations may include cultural transformation initiatives, existing practices aligned with COBIT 2019 objectives, and effective change management strategies. The research will investigate the following research inquiries: • By utilizing the COBIT 2019 design toolkit, what is the level of DevOps maturity within the company according to COBIT 2019? • What recommendations are required to enhance the maturity level in DevOps? 2. Literature Review Several practitioners have conducted studies on the utilization of COBIT as a reference for evaluating the maturity level of IT management. In the first place, Utomo's research, which assessed the maturity level of the Small Medium Enterprise (SME) focus area using COBIT 2019 at a university, revealed that 21 out of 40 governance/management objectives (52.5%) were chosen as priorities for the governance system at Campus A. These objectives were at an early stage, indicating a capability score of 1.03 out of a potential maximum of 5 points. Most of the processes were performed ad hoc and were not well organized or documented. Moreover, the research revealed that the design factors within COBIT 2019 can effectively cater to the specific attributes of SMEs, including their inclination towards operational and reactive approaches, limitations in resources and capabilities, as well as challenges related to project management and contract management. The incorporation of these design factors has influenced the customization of governance systems to align more effectively with the context of SMEs (Utomo et al., 2022). In the research conducted by Lestari regarding IT Governance for DevOps in an e-Marketplace company using COBIT 2019, it is stated that through data analysis from questionnaires, interviews, and expert assessments, six important IT processes were identified, namely APO03, APO04, BAI04, BAI06, BAI11, and DSS03, with a target capability level of 4. The implementation of IT governance in e- Marketplace companies in Indonesia needs to consider these design factors. Additionally, the use of the DevOps approach in e-Marketplace system development can enhance speed in creating systems and services, as well as prioritize automation in the provided solutions (Lestari et al., 2022).The authors assessed that the maturity assessment for the DevOps focus area is not yet optimal as the assessment was only conducted for 6 processes, while according to ISACA, there are 26 processes related to DevOps (ISACA, 2021). Another study conducted by Zarour regarding maturity models stated that most maturity models have 5 levels following CMM or CMMI, some have 4 levels, and one model has 10 levels (Zarour et al., 2019). Their research indicates that the use of DevOps maturity models is still very limited and has not been widely adopted by various organizations, raising questions about the validity of these models. Therefore, further research and empirical work are needed to test and validate the proposed DevOps maturity models. Another observation regarding the implementation of various DevOps maturity models is that no researchers have documented the assessment methods adopted in academic publications. The authors contend that this is inadequate. The developed assessment methods should also be published to enable their use or improvement by other researchers. In a study conducted by Mohamed (Mohamed, 2016) on the implementation of the DevOps Maturity Calculator (DOCM) in organizations that have adopted DevOps, the DOCM enables 928 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate organizations to integrate the DevOps maturity model into their business processes to meet client demands by measuring and assessing the level of DevOps maturity using a set of capabilities/criteria. The main objective of this work is essentially aimed at standardizing DevOps processes/best practices that will be adopted by a majority of current IT organizations to meet market demands. According to (Jabbari et al., 2016) assessing maturity levels is an important aspect of implementing DevOps practices in organizations. There are several frameworks used to evaluate DevOps maturity levels, including COBIT 2019 and ITIL v3. COBIT 2019 serves as a framework employed to oversee and regulate information technology within organizations. In the context of assessing DevOps maturity levels, COBIT 2019 can be used as a reference to measure the extent to which organizations have adopted DevOps practices. COBIT 2019 provides a maturity model consisting of 5 levels, where each level indicates the organization's maturity in implementing DevOps practices. By utilizing COBIT 2019, organizations can assess their DevOps maturity levels and identify areas that need improvement. 3. Methodology To answer the research questions, the researchers applied a structured approach based on the Design Science Research Methodology (DSRM) in Information Systems research (Peffers et al., 2018). DSRM focuses on developing specific artifacts to enhance the effectiveness of Information Systems and can accommodate various processes with an outcome-based approach. The artifact in this research is a customized maturity assessment system for DevOps, which is applied in a case study. Case studies allow researchers to examine a particular area more holistically and in-depth through empirical and real-life cases (Ebneyamini & Sadeghi Moghadam, 2018) despite potential criticisms regarding reliability and scientific rigor. The researchers conducted a collaborative workshop with the critical management team to obtain reliable and accurate information. The workshop engaged participants from the development and operations teams to gather and analyze data pertaining to current DevOps practices. 3.1. Research Steps • Identify the Objectives of the Proposed Solution: The main objective of this research is to assess the maturity level of DevOps in the organization using the COBIT 2019 framework. The researchers utilize theoretical concepts related to DevOps and relevant practices, along with the COBIT 2019 framework, to design a suitable maturity assessment and formulate appropriate recommendations. • Design and Development: The researchers organized a workshop involving the DevOps team in the organization. The workshop comprised of representatives from the development and operations teams, with the purpose of gathering and analyzing data concerning the prevailing DevOps practices. • Data Collection: The researchers employed a mix of qualitative and quantitative methodologies to acquire relevant information. The qualitative approach involved interviews, observations, and document examinations. Initial discussions were conducted to better understand the existing DevOps practices and identify areas that required evaluation. Subsequently, workshop sessions were conducted to assess the maturity level of each area using COBIT 2019 as a guide. • Data Analysis: The gathered data underwent analysis utilizing pre-established methods. The researchers used the COBIT 2019 framework to assess the maturity level of DevOps based on the obtained data. The analysis provided information about the organization's DevOps maturity level and identified areas that needed improvement. 929 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate • Presentation: After analyzing the data and identifying the DevOps maturity level, the researchers prepared a presentation to communicate the findings to the management team. The presentation included recommendations for enhancing the DevOps maturity level following the COBIT 2019 framework. • Evaluation: The research findings were presented to the management team to obtain feedback on the feasibility of implementing the proposed recommendations and identify any necessary adjustments. • Communication: The evaluation results of the DevOps maturity were documented in a report and communicated to the relevant management team for decision-making and appropriate follow-up actions. By following these steps, the Author utilized COBIT 2019 as a guide to evaluate and improve the maturity level of DevOps within the organization. 4. Result and Discussion PT XYZ is a state-owned enterprise operating in the mining sector. As per regulatory requirements, it needs to report its IT management's annual maturity level assessment using the COBIT governance framework. However, the regular COBIT reporting process is perceived as burdensome, mainly when conducting assessments for all processes. This is due to multiple stakeholders' involvement, the evidence preparation, and the high commitment required from all parties involved. In addition, PT XYZ is currently facing challenges in implementing the DevOps methodology within its Development and Operations environment. PT XYZ recognizes the need for effective IT governance to support better IT service delivery. Adopting the DevOps concept provides a viable solution to improve efficiency and foster collaboration between the development and operations teams. To measure the maturity level of DevOps, PT XYZ will leverage the COBIT 2019 framework. By utilizing COBIT 2019, the company can identify relevant domains and objectives in DevOps practices and build a governance system that aligns with its specific needs and goals. This research aims to examine the implementation of DevOps in PT XYZ, evaluate the achieved maturity level, and provide recommendations for enhancing the company's DevOps maturity. In conducting the maturity level assessment using COBIT 2019, the first step is to assign weights to the design factors provided by COBIT (Information Systems Audit and Control Association., 2018). These design factors are critical for designing the company's governance system to utilize Information and Technology (I&T) effectively. COBIT 2019 delineates 11 design factors that encompass Enterprise Strategy, Enterprise Goals, Risk Profile, I&T Related Issues, Threat Landscape, Compliance Requirements, Role of IT, Sourcing Model for IT, IT Implementation Method, Technology Adoption Strategy, and Enterprise Size. For measuring the maturity level, COBIT 2019 aligns with the Capability Maturity Model Integration (CMMI) (Information Systems Audit and Control Association., 2018). In COBIT 2019, alongside capability levels related to process areas, maturity levels are also included, representing a set or group of process areas. Both maturity and capability levels in COBIT 2019 use the same scale, ranging from 0 to 5, where higher levels indicate better maturity (ISACA, 2018). 930 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate Figure 1. Process capability level (Information Systems Audit and Control Association, 2018) COBIT offers capability levels for all process activities and delivers precise definitions of the processes and activities necessary to attain various capability levels. The intended levels are as follows: • Fully - Capability level achieved >85% • Largely - Capability level achieved >50% - 85% • Partially - Capability level achieved >15% - 50% • Not - Capability level achieved <15% In addition, COBIT 2019 also defines maturity levels as performance measures at the focus area level, as shown in the following diagram. Maturity levels are associated with the focus area, and a specific maturity level is achieved when all processes within the focus area reach a particular level of capability. • Incomplete: Processes may or may not be completed to achieve governance and management goals in the focus area. • Initial: Processes have been initiated, but the objectives and intentions of the focus area have not been fully achieved. • Managed: Planning and performance measurement is carried out, although not yet in a standardized manner. • Defined: Processes are implemented consistently across the organization. • Quantitative: The organization is driven by data, with a focus on quantitative performance improvement. • Optimizing: The organization focuses on continuous improvement. 931 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate Figure 2. Maturity level (Information Systems Audit and Control Association, 2018) 4.1. Maturity Assessment for DevOps Context In this stage, the Author will assess the maturity level of the processes in the state-owned company where this research is conducted using the Process Assessment Model (PAM) of COBIT, focusing on DevOps (ISACA, 2013). COBIT 2019 publication provides information about the DevOps focus area, comprising 26 processes (ISACA, 2021). These processes were assessed based on the evidence prepared by PT XYZ, including policies, procedures, work instructions, and other records provided during the assessment. The following table displays the outcomes of the capability level assessment for these 26 processes. Table 1. Assessment result. No. Processes Capability level 1 APO01 managed I&T management framework 3 2 APO02 managed strategy 3 3 APO03 managed enterprise architecture 2 4 APO04 managed innovation 2 5 APO05 managed portfolio 3 6 APO07 managed human resources 3 7 APO08 managed relationships 3 8 APO11 managed quality 3 9 APO13 managed security 4 10 BAI01 managed programs 2 11 BAI02 managed requirements Definition 4 12 BAI03 managed solutions identification and build 4 13 BAI04 managed availability and capacity 3 932 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate No. Processes Capability level 14 BAI06 Managed IT changes 3 15 BAI07 managed it change acceptance and transitioning 3 16 BAI08 managed knowledge 3 17 BAI10 managed configuration 2 18 BAI11 managed projects 4 19 DSS01 managed operations 3 20 DSS02 managed service requests and incidents 4 21 DSS03 managed problems 2 22 DSS04 managed continuity 3 23 DSS05 managed security services 4 24 DSS06 managed business process controls 4 25 MEA01 managed performance and conformance monitoring 4 26 MEA02 managed system of internal control 4 Based on the assessment results of the capability level for each process, the maturity level for the DevOps focus area of PT XYZ is obtained, as shown in the following figure. Figure 3. Maturity level result. Based on the data above, it is known that out of the 26 processes within the DevOps focus area, they have been measured and resulted in a maturity level of 3.15. There are four processes with a capability level of 2, 13 with a capability level of 3, and 9 with a capability level of 4. The value of 3.15 indicates 933 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate that the DevOps maturity level of PT XYZ falls under the 'Defined' category, meaning that the processes within the DevOps focus area have been implemented consistently within the PT XYZ environment. 5. Recommendation Based on the results of the maturity level assessment conducted, several recommendations for future improvements are as follows: 1. The company needs to prioritize the development of Enterprise Architecture (EA) according to the roadmap defined in the IT Master Plan (Wasiuk et al., 2021). In this process, the company should ensure that the high-level architecture is specifically accommodated in the EA document. 2. The company needs to develop an integrated EA management procedure. The company should consider the necessary steps to build, manage, and maintain EA effectively in this procedure. 3. The company should develop clear procedures for innovation management. These procedures should include steps to identify, evaluate, and implement innovations. It is also important to consider cross- functional collaboration in the innovation management process. 4. The company needs to develop specific procedures for program management. A clear understanding of the differences between programs and projects should be obtained and explained in these procedures. The company should ensure the existing procedures cover the necessary steps to manage programs effectively. 5. The company should develop specific procedures for problem management. Gaining a comprehension of the distinction between problems and incidents (Gehrmann, 2017). The implementation of the problem management module in IT Service Management (ITSM) tools should be considered. This will allow better records of problems and facilitate a proactive approach to addressing issues. 6. The company must identify the competencies required for each existing function and conduct competency assessments for the personnel involved. This will ensure that personnel responsible for specific functions or processes have the competencies required by the established requirements. 7. The company needs to develop clear performance measures for each process within the DevOps focus area. Existing Key Performance Indicators (KPIs) should be expanded to cover all related processes. Performance measures should go beyond just the achievement of Service Level Agreements (SLAs), the realization of IT work plans, and personnel training. This will help the company monitor and improve process performance effectively. 8. The company should implement a formal approach to continual improvement in every process included in the DevOps focus area (Value et al., n.d.). The company must identify areas that require improvement and consistently implement relevant measures for improvement. The execution of processes should involve the identification of potential improvements and be managed accordingly. Concrete improvement recommendations for the future should also accompany regular reporting. 6. Conclusions The case study research has reached its outcomes, and the researcher can conclude that the study has successfully achieved its objectives and benefits. This research has: • Conducted an assessment of the maturity level for the DevOps focus area using the Process Assessment Model (PAM) as a measurement tool. • Performed an analysis of the maturity level gaps in the DevOps focus area compared to the targeted maturity level. • Presented the analysis results and provided improvement recommendations for the processes within the DevOps focus area. • It generated a high-level process model or IT framework for IT governance in the researched company, focusing on the managed area of DevOps, which consists of 26 processes within it. 934 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 4: 926-934, 2024 DOI: 10.55214/25768484.v8i4.1472 © 2024 by the authors; licensee Learning Gate Copyright: © 2024 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). References Ebneyamini, S., & Sadeghi Moghadam, M. R. (2018). Toward Developing a Framework for Conducting Case Study Research. International Journal of Qualitative Methods, 17(1). https://doi.org/10.1177/1609406918817954 Gehrmann, M. (2017). Combining Itil, Cobit and Iso/Iec 27002 for Structuring Comprehensive Information Technology for Management in Organizations. Navus - Revista de Gestão e Tecnologia, 53–65. https://doi.org/10.22279/navus.2012.v2n2.p53-65.77 Information Systems Audit and Control Association. (2018). COBIT 2019 Framework Governance and Management Objectives. Information Systems Audit and Control Association. (2018). COBIT® 2019 Framework : introduction and methodology. ISACA. (2013). Process Assessment Model (PAM): Using COBIT 5. In Isaca. ISACA. (2018). Implementing and Optimizing an Information and Technology Governance Solution Personal Copy of Madalin Bratu (ISACA ID: 1283013). http://linkd.in/ISACAOfficial ISACA. (2021). COBIT Focus Area: DevOps. www.isaca.org Jabbari, R., Ali, N. Bin, Petersen, K., & Tanveer, B. (2016). What is DevOps? A systematic mapping study on definitions and practices. ACM International Conference Proceeding Series, 24-May-2016. https://doi.org/10.1145/2962695.2962707 Lestari, M., Iriani, A., & Hendry, H. (2022). Information Technology Governance Design in DevOps-Based E-Marketplace Companies Using COBIT 2019 Framework. INTENSIF: Jurnal Ilmiah Penelitian Dan Penerapan Teknologi Sistem Informasi, 6(2), 233–252. https://doi.org/10.29407/intensif.v6i2.18104 Mohamed, S. I. (2016). DevOps Maturity Calculator DOMC -Value oriented approach. International Journal of Engineering Research & Science, 2(2), 2395–6992. Peffers, K., Tuunanen, T., & Niehaves, B. (2018). Design science research genres: introduction to the special issue on exemplars and criteria for applicable design science research. In European Journal of Information Systems (Vol. 27, Issue 2, pp. 129– 139). Taylor and Francis Ltd. https://doi.org/10.1080/0960085X.2018.1458066 Ramaj, X., Sánchez-Gordón, M., Gkioulos, V., Chockalingam, S., & Colomo-Palacios, R. (2022). Holding on to Compliance While Adopting DevSecOps: An SLR. In Electronics (Switzerland) (Vol. 11, Issue 22). MDPI. https://doi.org/10.3390/electronics11223707 Utomo, D., Wijaya, M., & Tri Maretta Sagala, N. (2022). Leveraging COBIT 2019 to Implement IT Governance in SME Context: A Case Study of Higher Education in Campus A. In CommIT Journal (Vol. 16, Issue 2). Value, D., Two, F., Of, S., & Same, T. (n.d.). DevOps & ITSM : Defining Value From Two Sides Of The Same Coin. Wasiuk, T., Pol, F., & Lim, C. (2021). Factors Influencing Business IT Alignment. International Journal of Smart Business and Technology, 9(1), 1–12. https://doi.org/10.21742/ijsbt.2021.9.1.01 Zarour, M., Alhammad, N., Alenezi, M., & Alsarayrah, K. (2019). A research on DevOps maturity models. International Journal of Recent Technology and Engineering, 8(3), 4854–4862. https://doi.org/10.35940/ijrte.C6888.098319 https://creativecommons.org/licenses/by/4.0/