Engineering, Technology & Applied Science Research Vol. 7, No. 6, 2017, 2189-2193 2189 www.etasr.com Saeed et al.: The Impact of Capability Maturity Model Integration on Return on Investment in IT … The Impact of Capability Maturity Model Integration on Return on Investment in IT Industry: An Exploratory Case Study Anum Saeed COMSATS Institute of Information Technology Islamabad, Pakistan anum.saeed@comsats.edu.pk Raja Sher Afgun Usmani DCS&SE International Islamic University Islamabad, Pakistan sher.afgun@iiu.edu.pk Hina Akram COMSATS Institute of Information Technology Islamabad, Pakistan hina.akram20@gmail.com Syed M. Saqlain DCS&SE International Islamic University Islamabad, Pakistan syed.saqlain@iiu.edu.pk Anwar Ghani DCS&SE International Islamic University Islamabad, Pakistan anwar.ghani@iiu.edu.pk Abstract—Capability Maturity Model Integration (CMMI) is a framework known for improving the quality of processes on consistent basis. This paper makes an attempt to explore, analyze and describe the impact of CMMI in terms of return on investment (ROI) for the IT industry. An IT organization who has acquired a maturity level of CMMI 2 is taken as case study. At first, in reference to the level of applied CMMI, a thorough study is completed. Afterwards the luxuries and limitations are studied and explored with reference to organizations applying the monetary terms using ROI framework. Finally, the presented research highlights the key benefits and difficulties in switching to CMMI from conventional quality assurance. Keywords-CMMI; return on investment; ROI; IT; quality assurance I. INTRODUCTION IT industry today has to deal with increased direct customer interaction as more services of an organization are exposed to the customer directly. This is one of the biggest factors driving IT towards business goals such as customer satisfaction, performance, increasing security and decreasing time to market their products. In order to deliver such objectives, organizations are rapidly reviving themselves by spending more on Quality Assurance (QA) and testing. As of 2016-2017 organizations all over the world are spending 31% of their budget in addressing the quality problems [1]. In fact this year was proved to be different from the proceeding ones as the budget dropped from 35% to 31%. Before that, the budget of QA and testing was raised from 18% to 35%. However despite this year’s reduction, there is an overall prediction that QA and Testing budget will increase to 40% in 2019 [1]. To improve the quality of processes and control the cost of QA and testing, the organizations are adopting relative certifications and trainings. The Capability Maturity Model Integration (CMMI) is one the most popular training and certification programs adopted by more than 5,000 businesses from over 70 countries including United States, Germany, China, Italy, India, Australia, Pakistan etc. Many international organizations and companies demand CMMI maturity to grant contracts as CMMI makes sure that organization processes are of international standard, hence CMMI helps an organization to compete on global scale. Some of the salient advantages of CMMI are consistency, cost reduction, self-improvement and process improvement. The organizations that decide to improve the processes have to pay a significant investment of time and money. A variety of concerns are addressed by CMMI by ensuring process improvement through trustworthy manners. One of the better ways to present financial value is through Return on Investment (ROI) [2, 3]. This research highlights how to improve and map efforts to the corporate objectives and expectations regarding the effects of CMMI adoption. The ROI will be tracked on an ongoing basis. At the same time intermediate indicators like customer satisfaction, etc. will be tracked to determine the impact of improvement activities. ROI calculation is used along with other approaches to develop a business case for a given proposal. The key objective of this research is to present a report containing results about the impact of CMMI and also to explain the continuity and future work in this spot. This study specifies and signifies that executing CMMI practices for quality over conventional quality practices is beneficial and the cost spent on obtaining/applying CMMI has break-even with benefits. Furthermore, such information can be made part of quantitative management and be used as an input for improving processes and technologies. The training materials, appraisal methods and process improvement models used in this research are collected via Product Suite of CMMI. Benchmark versions Engineering, Technology & Applied Science Research Vol. 7, No. 6, 2017, 2189-2193 2190 www.etasr.com Saeed et al.: The Impact of Capability Maturity Model Integration on Return on Investment in IT … of the models and appraisal methods have been introduced. The use of CMM for scientific data management has been also proposed [5, 6]. II. LITERATURE REVIEW Several researches have shown numerous advantages of process maturity of an organization by using different assessment approaches [7-11]. An empirical study was carried out by authors in [12] to find out the effects of CMM on project performance and software quality in Information Systems (IS). The authors claimed that IS implementation strategies are related to CMM levels and higher CMM levels relate to higher software quality and project performance. Authors in [13] explored benefits of CMM such as rework, schedule, productivity, defects, error defection effectiveness, and ROI from the review of seventeen published articles. They concluded that investment in CMM programs leads to enhanced software development and maintenance. Process quality can be improved by adopting CMMI models [14]. In [15], author isolated the effort of process maturity effects versus other factors effects through using 161-project samples. Software development efforts can be reduced from 4% to 11% by raising organization to the next maturity level, but the percentage of reduction varies in each level of CMMI. Return on Investment [16], is an output observation when an organization implements some processes. The cost savings is realized by comparing it with the invested cost. The amount saved as a result of process improvement divided by the cost of process improvement is defined as improvement benefit index. The cost of improvement also caters the cost of personnel within a process improvement method such as consultant fees, purchasing tools, and the activities for improving management. The customer satisfaction is not easily quantified within the cost. With the similar procedure, the ROI can be calculated against each Key Process Area (KPA) of CMMI. According to authors in [17], business managers make their decisions on the basis of ROI as it is a highly accepted metric representing adaptability and simplicity. ROI can be calculated by using the following relation: ROI= (GI-COI)/COI, where GI= Gain from investment COI=Cost of investment If an investment has a negative ROI then usually it is not used. According to [18] in the IT sector, the ROI measures for evaluation are difficult to calculate by the management. Only 20% of organizations have formal ROI measures while 25% have plans to adopt it in the next 12 months. Literature for cost reduction and impact on schedule using CMM is discussed below. According to Boeing, Australia [19] the average cost to fix defects was reduced to 33%. Reduction in unit software cost was 20% according to Lockheed Martin [19]. 10% reduction in overall cost per maturity occurred according to Northrop Grumman [19]. Boeing reported 50% reduction in the release, in rework following test up to 60% reduction. General Motors reported a rise from 50% to 95% when number of milestones met. 25-30% productivity raised within three years [20]. Lockheed Martin reported 50% reduction of software defects and a 55% increase in award fees. Implementing CMMI process improves schedule and budget predictability, cycle time, productivity, quality, customer satisfaction, and ROI. CMMI process improvement may be established in various ways. For adopting CMMI Product Suite, several organizations have integrated, changed and updated the current processes with the CMMI processes, while some have established new processes. Some organizations have expanded the scope of their improvement efforts by integrating software, hardware and related disciplines with the CMMI model to their existing models. In some cases, few organizations do not adopt CMMI. However, they have established a process which is emphasized differently than the Software Capability Maturity Model (SWCMM), but the outcome shows the significance of the finest practices that are articulated in CMMI models. However, these organizations cannot be accredited as CMMI adopted. Similarly, the impact of CMMI-based process improvement can be used to demonstrate many possible measures of performance. A high- level description of the CMMI impact based process improvement is shown in Figure 1. In the upper left, costs of process improvement are illustrated. In the center, box process capability and organizational maturity are depicted, and the benefits of process improvement are illustrated in the box on the right. As shown, in order to calculate ROI or related measures the costs and benefits are combined [21]. III. LIMITATIONS Consistently measuring ROI is tough since organizations tend to refrain from sharing their financial data. The study concentrates on one specific software organization in Pakistan, so the facts provided are specific to that organization. Fig. 1. High-Level Model of CMMI Impact IV. METHODOLOGY CMMI is one of the ways to implement the quality plans and is generally used as a standard for software quality. The best way to plan quality is to implement CMMI. The quality is achieved by following and implementing CMMI’s defined rules and processes. Based on CMMI standard, the Center for Advanced Research in Engineering (CARE) needs to create a Quality Plan that fulfills the requirement of CMMI. Engineering, Technology & Applied Science Research Vol. 7, No. 6, 2017, 2189-2193 2191 www.etasr.com Saeed et al.: The Impact of Capability Maturity Model Integration on Return on Investment in IT … A. Introduction of Organization An engineering organization is taken as a case study. It has a history of good reputation by receiving acknowledgements in the form of international awards. It is a certified CMMI compliant organization with an engineering team of more than fifty employees and more than fifty employees as supporting staff. In the areas of emerging technologies, the organization offers an extensive list of products, expertise and design services. It stands at the forefront of the technology by focusing on research and development. The organization got flourished through a well-trained, creative, diverse and outstanding team responsible for outstanding products and system design portfolio. The organization achieved CMMI level 2 by implementing all 7 KPAs of Level 2 in two years. A brief summary of CMMI related implementations are:  Process definition by Quality Engineering department.  Training for managers and engineers regarding CMMI.  Developed in-house software for configuration management, which also worked with Concurrent Versions System (CVS) and Visual Source Safe (VSS).  Developed in-house software for reporting.  Hired people for configuration management.  Deployed special focus on documentation and task trackers. V. RESULTS A. Cost This category covers the areas where organization reports reduction in the cost of the work product, process and general savings to model-based improvement at various stages, intermediate and final. The summary is shown in Table I. TABLE I. SUMMARY OF BENEFITS AND IMPACT: RETURN ON COST Evaluation Result Average cost to fix defects 20% Reduction Unit testing Process 7% Reduction Overhead rate 2% decrease Average cost variance 15% Reduction in expenses in first six months 2 million B. Schedule The two important aspects i.e., schedule predictability improvement and reduction in time to complete tasks, are covered under this category. Productivity is determined by the quantity of work done over a period of time. The summary is shown in Table II. C. Quality The quality improvement is measured with reference to the reduction in the number of defects. Pre-test rework is decreased throughout the duration of a project, which helps in satisfying the customer while improving the project quality. The main objective of the organization is to satisfy its customers by minimizing the defects through their identification at early stages. To achieve this goal, formal inspections of CMMI are quite helpful. The summary of benefits and impact of using CMMI standard with reference to quality is presented in Table III. TABLE II. SUMMARY OF BENEFITS AND IMPACT: SCHEDULE Evaluation Result Reduction in the time required to turn around release 25% Reduction in work due to pre and post test 60% Increase in productivity 20% Improvement in on-time delivery of project 15% Improved and stabilized schedule YES TABLE III. SUMMARY OF BENEFITS AND IMPACT: QUALITY Evaluation Result Defects found in fielded systems 1% Increase in focus on quality by developers YES Reduction in severity of post-release defects YES Improvement in Quality of a code YES D. Customer Satisfaction Quality improvement leads towards customer satisfaction. Customer satisfaction can also be seen by measuring the process improvement benefits. The summary of customer satisfaction achieved through CMMI is given in table IV. TABLE IV. SUMMARY OF BENEFITS AND IMPACT: CUSTOMER SATISFACTION Evaluation Result Increase in award fees: 20% Received 58% customer award fees CMMI Reduction in severity of post-release defects? YES Improvement in Quality of a code? YES E. Return on Investment (ROI) Return on Investment (ROI) may be calculated through a number of ways yet none of them is defined as official standard. Some of these are as: • Benefit - cost ratio (BCR) • Net present value (NPV) • Internal rate of return (IRR) • Payback period (PP), breakeven point (BEP) ratio ROI of an investment (capital) is the ratio of “Total Earnings before interest and taxes” and “average invested capital”. However, for calculating ROI for a process improvement project, the computation used is: Return on Investment for a Process Improvement Project = (Benefits from Process Improvement – Cost (investment) incurred towards undertaking process improvement). Engineering, Technology & Applied Science Research Vol. 7, No. 6, 2017, 2189-2193 2192 www.etasr.com Saeed et al.: The Impact of Capability Maturity Model Integration on Return on Investment in IT … 1) Past Period Initial investment: 285, 000,000, netprofit: 25,000,000. ROI:10% 2) CMMI Impact Initial investment: 40,000,000, net profit: 75,000,000. ROI:18% TABLE V. SUMMARY OF BENEFITS AND IMPACT: ROI Results Model 4:1 ROI of quality activities CMMI 7:1 defects calculated per hour and time spent in preventing errors CMMI Earlier defects detection and better project control implemented CMMI F. Benefits and Cost ratio ROI can be calculated as simply the ratio of enjoyed benefits by a certain effort to the deserved cost. Pros:  Easy to compute  Provides ways to compare the projects Cons  Time value of money (TVM) is not incorporated in ROI  No approach to identify the dollar enormity of ROI  Cannot predict the time to achieve the desired results. Fig. 2. Impact of CMMI on ROI TABLE VI. SUMMARY OF BENEFITS AND IMPACT OF CMMI Before Applying CMMI After Applying CMMI (Year 0) After Applying CMMI (Year 1) Total Cost 3 million 1 million 1 million 5 million Benefits 0 5 million 5 million 10 million Return 10% 18% 22% 15 million Fig. 3. Summary Benefits and Impact of CMMI Figure 4 shows the rework have gone out of boundaries at periods 1, 3, 4, 5, and 6, while at other points everything is well managed. Fig. 4. Summary Benefits and Impact of CMMI (Incorporating Boundary Conditions) VI. CONCLUSION Based on the discussion with project managers and team leaders, it was concluded that customer satisfaction can only be achieved by managing and incorporating their requirements into to the product. The following list summarizes the benefits of CMMI implementation.  The overall product cost along with finding and fixing the defects gets lowered.  There is considerable reduction in time to complete the desired projects.  The number of product defects is reduced, hence the quality gets improved.  Rising of award fees demonstrates customer satisfaction through earlier and continuous delivery of valuable software  This case study claims a positive ROI from CMMI- based process improvement. This report validates that an organization benefits from CMMI, if it works hard at process improvement. It is hoped Engineering, Technology & Applied Science Research Vol. 7, No. 6, 2017, 2189-2193 2193 www.etasr.com Saeed et al.: The Impact of Capability Maturity Model Integration on Return on Investment in IT … that from the current review, the organizations wishing to adopt CMMI can get useful information regarding its impact on ROI. VII. RECOMMENDATION For more improvement in the organizational processes and longtime success through customer satisfaction, organizations should take steps towards level 3, 4 and 5. OPD an OPF will provide a base to organization improving, better managing their processes and aligning their organizational vision with the work. Upgrading their work to a higher level will be most helpful for them. REFERENCES [1] Capgemini. World quality report 2016-17, Available at: https://www. capgemini.com/thought-leadership/world-quality-report-2016-17, 2017 [2] M. C. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, “Capability maturity model, version 1.1”, IEEE software, Vol. 10, No. 4, pp. 18–27, 1993 [3] M. Agrawal, K. Chari, “Software effort, quality, and cycle time: A study of CMMI level 5 projects”, IEEE Transactions on software engineering, Vol. 33, No. 3, pp. 145–156, 2007 [4] M. Diaz, J. King, “How cmm impacts quality, productivity, rework, and the bottom line”, The Journal of Defense Software Engineering, 2002 [5] K. Crowston, J. Qin, “A capability maturity model for scientific data management: Evidence from the literature”, Proceedings of the American Society for Information Science and Technology, Vol. 48, No. 1, pp. 1–9, 2011 [6] R. Bate, An integrated product development capability maturity model, CMU/SEI-97-MM-001, Software Engineering Institute, Carnegie Mellon University, 1997 [7] K. Butler, “The Economic Benefits of Software Process Improvement”, Proceedings of Crosstalk, Hill AFB, USA, pp. 14-17, 1995 [8] J. Herbsleb, A. Carleton, J. Rozum, J. Siegel, D. Zubrow, Benefits of CMM-Based Software Process Improvement: Initial Results, CMU/SEI- 94-TR-13, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, 1994 [9] S. Humphrey, R. Snyder, R. Willis, “Software Process Improvement at Hughes Aircraft”, IEEE Software, Vol. 8, No. 4, pp. 11-23, 1991 [10] T. McGibbon, A Business Case for Software Process Improvement, Final Report, Contract Number F30602-92-C-0158, Data and Analysis Center for Software, Kaman Sciences Corporation, New York, 1996 [11] H. Wohlwend, S. Rosenbaum, “Schlumberger’s Software Improvement Program”, IEEE Transactions on Software Engineering, Vol. 20, No. 11, pp. 833-839, 1994 [12] H. Girish, J. James, G. Klein G., “Software Quality and IS Project Performance Improvements from Software Development Process Maturity and IS Implementation Strategies”, Journal of Systems and Software, Vol. 80, No. 4, pp. 616-627, 2007 [13] D. Galin, M. Avrahami, “Are CMM Program Investments Beneficial? Analyzing Past Studies”, Software IEEE, Vol. 23, No. 6, pp. 81-87, 2006 [14] M. Diaz, J. King, “How CMM Impacts Quality, Productivity, Rework, and the Bottom Line”, The Journal of Defense Software Engineering, Vol. 15, No. 3, pp. 9-14, 2003 [15] B. Clark, “Quantifying the Effects of Process Improvement on Effort”, IEEE Software, Vol. 17, No. 6, pp. 65-70, 2000 [16] D. J. Paulish, A. D. Carleton, “Case studies of software-process- improvement measurement”, Computer, Vol. 27, No. 9, pp. 50–57, 1994 [17] K. El Emam, D. R. Goldenson, “An empirical review of software process assessments”, Advances in Computers, Vol. 53, pp. 319–423, 2000 [18] B. Violino, Measurin Return on Investment, Information Week, pp. 36– 44, June 30, 1997 [19] D. Goldenson, D. L. Gibson, Demonstrating the impact and benefits of CMMI: an update and preliminary results, Software Engineering Institute, Carnegie Mellon University, 2003 [20] L. G. Jones, A. L. Soule, Software process improvement and product line practice: CMMI and the framework for software product line practice, CMU/SEI-2002-TN-012, Software Engineering Institute, Carnegie Mellon University, 2002. [21] A. Q. Liu, “Motorola software group’s china center: Value added by CMMI”, Software Tech News, Vol. 10, No. 1, pp. 18–23, 2007