Journal of Software Engineering Research and Development, 2021, 9:15, DOI: 10.5753/jserd.2021.1944 This work is licensed under a Creative Commons Attribution 4.0 International License. Software process improvement programs: What are the pitfalls that lead to abandonment? Regina Albuquerque [ Pontifícia Universidade Católica do Paraná| regina.fabia@pucpr.br] Gleison Santos [ Universidade Federal do Estado do Rio de Janeiro | gleison.santos@uniriotec.br] Andreia Malucelli [ Pontifícia Universidade Católica do Paraná | malu@ppgia.pucpr.br] Sheila Reinehr [ Pontifícia Universidade Católica do Paraná | sheila.reinehr@pucpr.br] Abstract While many organizations successfully embrace and experience software process improvement (SPI) benefits, others abandon the effort before realizing the total potential result of an SPI initiative. Therefore, researchers' interest in understanding the reasons why software organizations that have a successful start in adopting SPI abandon improvement initiatives after evaluation has increased. Thus, this work aims to investigate how the abandonment of SPI programs based on maturity models occurs after the evaluation. The multiple case study method was used with eight organizations. Data were analyzed using Grounded Theory open and axial coding procedures. The results show that SPI initiatives failed because of internal factors (people, SPI project management, organizational aspects, and processes) and external factors to the organizational context (country economic crisis, outsourcing, governmental political influence, and external pressure from the client). As a contribution, we highlight the identification of these factors that organizations can use to learn about their initiatives and avoid pitfalls that can lead to the abandonment of SPI. Keywords: Software and its engineering, Software Quality, Software Process Improvement, Abandonment of Software Process Improvement 1 Introduction Software organizations operate in a highly competitive market that demands quality and productivity (CANEDO et al., 2019). In this sense, Software Process Improvement (SPI) aims to offer insights into the software process as it is used within organizations and, thus, lead to the implementation of changes to achieve specific objectives, such as increasing product quality or reducing cost and development time (Coleman et al., 2008). Several process improvement support models have gained ground in the software industry, such as CMMI-DEV (CMMI INSTITUTE, 2018) and ISO/IEC 33020 (ISO/IEC, 2015). In Brazil, where this research was conducted, the MPS.BR (Brazilian Program for Software Process Improvement) resulting model is primarily used. MPS.BR is a mobilizing, long-term program that aims to define software and service process improvement and assessment models targeting primarily micro-small and medium-sized enterprises to meet business needs (SOFTEX, 2020). The MR-MPS-SW (Brazilian Reference Model for Software Process Improvement) model is structured in seven evolving maturity levels. They are a combination of processes, which are based on ISO/IEC 12207 (ISO/IEC, 2017), and compatible with CMMI-DEV (CMMI INSTITUTE, 2018) and their capabilities, which are based on ISO/IEC 33020 (ISO/IEC, 2015). The maturity levels establish thresholds of process evolution that characterize improvement stages for SPI implementation in software organizations. The maturity evolution begins with Level G and progresses up to Level A (SOFTEX, 2020). To qualify their processes, organizations must undergo an official assessment, which is valid for three years. Previous studies have reported benefits such as higher customer satisfaction, cost reduction, greater predictability of costs and deadlines, and increased productivity and quality (Kalinowski et al., 2010). Until April 2021, 816 assessments had been successfully completed (http://www.softex.br/). Many organizations were assessed at the initial levels G (55%) and F (31%). Only 14% of the assessments are associated with the upper levels (Level E: 4%, Level C: 9%, and Level A: 1%), which signifies that progress occurs up to level F, in general. That suggests that most organizations either abandon their SPI programs or maintain compliance with the maturity level requirements without undertaking renewal appraisals. Therefore, an important question arises: if companies achieve benefits by improving software processes, why do they abandon SPI programs? Our previous research has pointed to organizational, human, and process-related issues (Albuquerque et al., 2018). Other research studies have sought to gather further information on maintaining process practitioners' participation after the appraisal period (Uskarci et al., 2017). Nalepa et al. (2019) and Fontana et al. (2015) have found a different way for organizations that use agile methods to mature. Understanding how companies continue to improve their processes after an appraisal is relevant to the software industry, which still faces challenges posed by time and budget constraints that may hinder SPI initiatives' continuation. Given this context, the aim of this study is to understand how abandonment occurs in SPI programs after a successful Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 assessment based on maturity models. To accomplish this objective, we conducted case studies in eight Brazilian software companies. Data were analyzed using open and axial coding procedures from Grounded Theory (Strauss and Corbin, 1998). SPI managers can use the results of this research to avoid the pitfalls that can lead to abandoning the SPI initiative. Results from four of these organizations were published in Albuquerque et al. (2020). The main contribution of the present paper is the confirmation that factors internal to the organization (Human, Organizational, SPI project, and processes) and factors external to the organization (the economic crisis of the country) when neglected can cause the abandonment of the SPI. In addition, new results emerged, such as lack of external demand for evaluation1, dissolution of the company, the fusion of companies, and adherence to agile methodologies. The paper is organized into seven sections besides this introduction: Section 2 presents the background; Section 3 describes the research method; Section 4 reports the results; Section 5 presents the discussion; Section 6 presents threats to validity; Section 7 presents the final considerations. 2 Related works Software Process Improvement (SPI) is an approach that has attracted the interest of software companies because it promises to increase quality and decrease costs and project deadlines (Coleman et al., 2008). While many organizations successfully adopt and experience the benefits of SPI (Kalinowski et al., 2010), others abandon the effort before realizing the potential of SPI benefits (Albuquerque et al., 2018). Therefore, there is an interest in understanding the reasons why these companies abandon these improvement initiatives. Almeida et al. (2011) have identified factors that can affect continued adherence to the software process in an organization, focusing on the software processes assessed using MR-MPS-SW as a basis. The results of their study were classified into four factors: technical factors, sociocultural factors, resources and, commitment. Besides, they have shown that project management processes are challenging to maintain in the routine of companies. Uskarci et al. (2017) sought to identify the problems of continuity and participation in software process improvement activities in two Level 3 CMMI-DEV companies in Turkey. They have identified higher submission rates of suggestions for improving the process when the assessment date is approaching and lower rates when the assessment is completed. Besides, the employees' participation in these activities and their prospects for process improvement are highly dependent on their role within the organization. The authors have identified greater involvement of employees in the quality group and process 1 In some parts of this text the term certification will be used meaning evaluation, specially in the transcriptions of the interviews. group. On the other hand, practitioners of the process are reluctant to suggest improvements in the process. Albuquerque et al. (2018) present a survey conducted in Brazil to identify which factors (based on a systematic literature review) can lead to SPI programs' maintenance or abandonment. The interviewees comprised specialists in SPI (consultants and appraisers of CMMI-DEV and MR-MPS- SW models). Results indicate that SPI programs continuation is positively influenced by human factors (motivation and acceptance; support, commitment, and involvement; technical and personal competencies), the SPI project itself (definition of strategies; resources; adequate external consultancy service), organizational factors (communication; goals; organizational structure; internal and external policies; return on investment and leadership), consultancy and processes. Albuquerque et al. (2019) investigated how organizations using agile methods evolved their processes after assessing the maturity model. The unit of analysis of the case study was four privately owned software organizations that have been assessed with the MR-MPS- SW model and that used agile methods. Results showed that companies using agile methods have difficulties in implementing SPI initiatives with maturity models. It was found that processes based on maturity models were partially abandoned and that project management practices are the most difficult to maintain, confirming the results found by Uskarci et al. (2017). According to Anastassiu et al. (2020), the resistance negatively affects SPI, both in implementation and maintenance. They conducted a qualitative study on the causes and effects of change resistance in SPI initiatives and procedures to mitigate resistance. They interviewed 21 professionals and specialists in improving software processes. The authors identified 32 causes of resistance, 16 effects, and 29 behaviors related to resistance to change. Among the results, it is worth highlighting the effects that resistance creates in SPI initiatives, were: EF01: Rejection of resistant members who boycott the process, EF02: The firing of members resistant to change and/or to follow the process, EF03: Demotivation of the process team due to the resistance of its executors, EF04: Compromised improvement project goals, EF05: Use of bypass solutions, EF06: Abandonment of the process, EF07: Real improvements are not achieved, EF08: Demotivation due to the difficulty in changing the culture, EF09: Skepticism due to the difficulty in changing culture, EF10: Resignation from employment because of the difficulty in changing culture, EF11: Inappropriate attitudes (rebellious and deceitful) by some of the leaders, EF12: Feeling of isolation in the organization, EF13: Submission by fear by middle management and executors of the process, EF14: Bad influence for new hires, EF15: One-off and non-continuous improvements and EF16: Fear of job loss. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 Although previous studies have provided information on the post-assessment phase, they have limitations for not addressing information regarding the abandonment of SPI. It is crucial for organizations interested in adopting SPI to know what causes can lead to SPI failure to avoid or mitigate these risks. For example, Almeida et al. (2011) and Uskarci et al. (2017) reported results from organizations with valid official assessments. In Albuquerque et al. (2018), the authors reported a survey with SPI specialists and Anastassiu et al. (2020) in a qualitative study with SPI specialists. Although these specialists' point of view is relevant, it is essential to conduct qualitative research to identify how human, organizational, SPI project, and process factors influence SPI initiatives' continuity in organizations from the organizations' point of view. Albuquerque et al. (2019) presented the difficulty of agile companies in sustaining SPI programs using maturity models. However, there is a lack of information about organizations' challenges with their overdue official assessments. Understanding this topic is essential to conduct qualitative research in different contexts and from the organizations' perspective. 3 Research Method This paper addresses the following research question: RQ: How does abandonment occur in software process improvement programs? To answer the question, we conducted a case study in eight software organizations. Yin (2017) states that when the research aims at answering a "how" question, a case study is a method that offers the response. In case studies, the definition of propositions guides data collection and analysis. They also help to accomplish the research objective. Based on the literature (Albuquerque et al. (2018), Almeida et al. (2011), Albuquerque et al. (2019), and Uskarci et al. (2017), the following propositions were defined: ▪ P1. There are human factors that influence the abandonment of the SPI program. ▪ P2. There are SPI design factors that influence the abandonment of the SPI program. ▪ P3. There are organizational factors that influence the abandonment of the SPI program. ▪ P4. There are process-related factors that influence the abandonment of the SPI program. 3.1 Context The analysis unit, also called a case, is a software organization evaluated by the MR-MPS-SW model and has not carried out new evaluations. An organization was considered to be abandoning SPI when they reported no longer using the processes (organizations 4 and 8) or partially using it (organizations 1, 2, 3, 5, 6, and 7). We carried out the case study in eight software organizations with different profiles, as shown in Table 1. Organizations of various sizes participated in this research, such as small (2 and 7), medium (3 and 8), large (1, 4, and 6), and micro-enterprise (5). Only organization 1 is from the public sector. Regarding the main activities, organizations 1, 4, and 8 maintain software products and develop custom software. Organizations 2, 3, 5, and 7 perform maintenance on software products. Organization 6 develops software and offers software services. Table 1. Profile of the studied companies. Org. Profile Data 1 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 300 Employees Public TIC Not Not G June 2016 2 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 40 Employees Private ERP product Not Yes F January 2017 3 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 80 Employees Private ERP product Yes Not C November 2018 4 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 100 Employees Private Custom/Embedded Software Not Yes E May 2018 5 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment 05 Employees Private ERP product Not Yes G November 2015 6 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 270 Employees Private Software factory/ services Yes Not C January 2020 7 Organization size Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment + 30 Employees Private ERP product Not Yes F August 2019 8 Organization size + 50 Employees Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 Origin of capital Main activity Participate in bidding Federal grant Maturity Level The validity of the assessment Private ERP product/ Software factory Yes Yes F September 2015 Only organizations 3, 6 e 8 participates in government bids. It is worth clarifying that in Brazil, the federal government launches bids to carry out software projects. Some of them require the company to have a valid assessment compliant with a quality model or standard. Therefore, a company that has a maturity model evaluation can achieve a higher score than its competitors. To incentivize organizations to improve their processes, SOFTEX has developed a business model that offered some financial support for organizations with less than 100 employees. Organizations that were interested in implementing the reference models of the MPS.BR program could have had financial support by MCT (Ministry of Science and Technology) or by SEBRAE (Support Service for Micro and Small Companies) (SOFTEX, 2020). Regarding the federal grant, organizations 2, 4, 5, 7, and 8 received this benefit. Table 1 also shows the MPS-SW maturity level that the organization accomplished in its last evaluation. The study was conducted with: 2 level G organizations (1 and 5), 3 level F organizations (2, 7, and 8), 2 level C organizations (3 and 6), and one level E organization (4). 3.2 Data collection We sent a letter of introduction to the organizations explaining the research objectives with a Non-disclosure Agreement (NDA) signed by the researchers for data collection. To obtain the vision of different software development roles, we interviewed people in management positions (sponsor, director, project manager, process improvement team, and quality assurance) and software engineers (analysts, developers, and testers). Table 2 shows the participants' profiles. Table 2. Profile of the participants. Org. Participants 1 1 sponsor 1 SPI manager 3 project managers, 1 coordinator of project managers 2 quality assurance analysts 4 analysts and developers (acting in both roles) 2 1 sponsor 1 project managers 1 development director 3 analysts and developers (acting in both roles) 3 1 quality assurance manager 4 1 process manager 5 1 sponsor 6 1 sponsor 1 human resources manager 7 1 sponsor 1 project managers 1 quality assurance manager 8 1 sponsor As shown in Table 2, in some organizations, due to high turnover, only one person who took part in the SPI initiative was still in the company to be interviewed. We built a semi-structured script to guide the interviews. The questionnaire consisted of two sets of questions: one to characterize the organization and interviewee profiles, and the other about SPI, aiming to gather information about the challenges faced after evaluating the company and the strategies to deal with these challenges. The second part also helped to obtain information about the processes considered challenging to continue after the assessment. The following questions were used as a semi-structured interview script to guide the researcher. It is worth noticing that the questions asked in the field were broader to allow higher data coverage and richer answers. The questions supported the researcher while conducting the semi- structured interview acting more as a checklist than a fixed route: Part 1 - Characterization questions ▪ Can you describe the organization in terms of business and culture? ▪ What position do you currently hold in the organization? ▪ How long have you worked in the organization? ▪ What is your academic background? Part 2 - Questions about SPI ▪ How is top management involved, and which support is offered to the SPI program? ▪ What is your perception of the involvement and support of the technical team in the SPI program? ▪ Is there an ongoing investment in training? Which trainings are offered? ▪ What is your perception of the involvement and support of the technical team in the SPI program? How have the improvement program activities changed your development activities? Are the activities easier or harder to work with? ▪ Is there a specific budget for the SPI project (hours, staff, infrastructure)? How is the SPI project structured in terms of infrastructure (environment and tools) and staff? ▪ How are changes in the organization's development process made? Who defines the process activities, and who determines how they are executed? How are the changes introduced in the projects? ▪ How did the consultant evaluate the company's previous process before defining the current process? How do you Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 evaluate external consultancy's performance during the improvement model's implementation period (hours of service, relationship, competence)? ▪ Is the company interested in renewing or evolving its maturity level? Why (not)? How is the SPI program aligned with the organization's strategic planning? - How are these business goals monitored in the organization? ▪ Is there a software engeneering process group (SEPG) to lead process improvements implementations? What is the composition of this group? How are the activities of the SEPG conducted (meetings, periodicity)? What is the degree of influence of this group on the company's other groups regarding knowledge, reputation, and relationships? ▪ How constant is the organization's project flow? How are the roles and responsibilities shared within the organization? Is turnover an issue in the organization? How is it avoided? ▪ How are the improvement project goals communicated to the employees? ▪ How is day-by-day communication performed in the SPI project? How are the results of the SPI project communicated to the employees? ▪ How are the processes used in the organization? Are they used in all areas and projects? ▪ Which processes are most challenging to maintain? Why? ▪ Which processes are more natural to maintain? Why? ▪ Are there performance indicators for the SPI project? ▪ How is the return on investment (ROI) of the SPI project measured (for instance, product quality, customer satisfaction, market expansion, estimates, cost, and term)? How are the process activities monitored (i.e., detection of nonconformities and their solution)? 3.3 Data analysis Yin (2017) guides the researcher to define the logic that links the data to the study's propositions and the criteria to interpret the results. In this research, we used the model proposed by (Reinehr et al., 2008) that defines Points of Analysis (PA), which supports concepts based on the literature review to evaluate whether a proposition is confirmed or not to answer the main research question. Table 3 shows the defined research propositions and related points of analysis. The propositions, as previously explained, are the statements of what the researchers expect to find in the field study, based on the previous literature. The points of analysis are the connection between data collected in the field and propositions analysis. The theoretical basis for constructing these research elements (propositions and points of analysis) was the background presented in section 2, a systematic literature review, and the survey carried out with SPI specialists presented in Albuquerque et al. (2018). The categories of critical factors for SPI maintenance were used (human, organizational, SPI project, and process) to define the propositions. To determine the points of analysis, we used the factors related to each category: ▪ human factors: motivation and acceptance, support, commitment, and involvement, technical competencies; ▪ organizational factors: goals, communication, organizational structure, internal and external policies, return on investment and leadership; ▪ the SPI project itself: definition of strategies, resources, appropriate external consultancy service, consultancy; and, ▪ processes factors: level of bureaucracy; measurement program for continuous improvement. Table 3. Propositions and points of analysis. Proposition P1. There are human factors that influence the abandonment of the SPI program PA.01: Training is offered for the qualification of the employees of the company. PA.02: There is support, commitment, and involvement of organization members. PA.03: The technical team members are motivated and willing to carry out the process activities. Proposition P2. There are SPI project factors that influence the abandonment of the improvement program. PA.04: Budget and resources are available for the SPI initiative. PA.05: There is a strategy to introduce changes in software processes. PA.06: Existence of an external consultancy with the ability and competence to implement a process compatible with company needs. Proposition P3. There are organizational factors that influence the abandonment of the improvement program. PA.07: Existence of a strategic plan that relates the SPI program to business goals achievement. PA.08: Leadership is available to support continuous process improvement. PA-09: There is an organizational structure favorable to the SPI program. PA-10: There are communication mechanisms for the dissemination of the SPI project. Proposition P4: There are process-related factors that influence the abandonment of the improvement program. PA.11: There is a non-bureaucratic process that meets the needs of the company. PA.12: There is a measurement program of continuous process improvement. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 We used Grounded Theory (Strauss; Corbin, 1998) open and axial coding procedures for qualitative analysis because it is a systematic analysis approach, which adds value in terms of academic rigor, providing validity in terms of traceability from the coding of the initial data to the final result of the analysis (O'Connor, 2012). We did not intend to create a theory using the interactive process of conducting interviews and then analyzing the data to guide the following interviews, oriented by Strauss and Corbin (1998). We did not achieve saturation as preconized by Colleman et al. (2008). All the interviews were recorded and then transcripted. We performed the analysis after all interviews were completed. The transcript was read (more than once by the first author) and analyzed with the support of the Atlas.TI tool. The first author performed the open coding activities, which is the microanalysis of the interviews. She analyzed each transcript line-by-line and created codes merged with existing codes as appropriate when new evidence data appeared. Memos were created to support the analysis (also considering the field notes). Then, the codes were grouped according to their properties, forming concepts that represent categories. Finally, the categories and subcategories were related to each other in the axial coding stage. All the analyses were reviewed and discussed by the other authors. Figure 1 shows how we identified the presence or the absence of a point of analysis in the interview excerpts and related them to the research propositions. As can be seen in Figure 1, we used codes that differentiate the encoding stages. In open coding, codes called types of findings were identified with an [A]. Codes from the axial coding cycle were grouped into Negative Factors [NF] and Positive Factors [PF]. Subsequently, these positive and negative factors were grouped into the category called analysis points [PA]. Figure 1. Extract of codes and citations related to PA.12 Monitoring. The example shows a Negative Factor [ND]. When the researcher asked: "How is the process monitored?" two participants answered, "No. This has not been done recently, because there is no professional to guarantee the quality" and "There is no charge for non-conformities in the process". Based on these statements, the code generated was "Lack of QA professional to guarantee the quality of the process". The same coding process was applied to the code "Lack of control and collection of process evidence" which is contrary evidence to the code "Monitoring the improvement process" which, in turn, is part of the point of analysis "PA.12_Measurement program". Later in the codification process, the analysis point mentioned above was related to "Proposition P4. Processes". During the analysis, new findings emerged from the data. These codes were called New Discovery [ND], with the ND code followed by a number. 4 Results 4.1 Analysis of individual cases The following sections present the description of the analysis of each case study, listing the points of analysis (PA), the new discoveries (with the ND code followed by a number), and the participants' quotes. In addition, we present the context of SPI in the implementation and maintenance period. 4.1.1 Organization 1 Implementation period. The reasons for the adoption of the maturity model were process improvement and market. The board appointed a team to work on the SPI project, providing training for a group of people who participated in the definition of MR-MPS-SW level G processes. At the beginning of the implementation, the SPI was disseminated through different communication means (lectures, training, e-mail, and intranet). Still, only the people directly involved with the group of processes were better informed. There was no hiring of consultants once people in the organization had experience implementing maturity models (PA.06). The quality assurance team monitored the process, and non-compliances were dealt with. The main difficulties were: failure in communication (as the organization is large, some people were uninformed), insufficient training, an overload of work due to the accumulation of functions, lack of human resources, bureaucracy in the process, and resistance to changes. Interviewee: Training (PA.01) and Communication (PA.10). "We feel that people are doing the projects; they take the templates and come to ask. But how do I do this? Will I attend the course? Because we feel this … that there is still a failure in the issue of communication, because there are more than 300 people in the development area, so there are many people who are not yet having this level of information." Maintenance period. Organization 1 reported no intention to evolve its maturity level because the development area remained immature in project management practices. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 The training (PA.01) did not cover the whole development area. The lack of support (PA.02) from top management to demand project coordinators to use the processes led process practitioners and the quality assurance team to lose motivation (PA.03). For example, the quality assurance team failed to monitor processes because managers did not take the corrective actions needed after quality assessments. The lack of human resources (PA.04) resulted in the outsourcing of the projects. There is an active process group that defined strategies to support SPI (PA.05). Outsourcing (ND.01) was a new aspect that emerged during the analysis. For managers, it is difficult to adhere to process methodology in outsourced projects. There was an attempt to mentor the outsourced company, but it did not work out due to the high turnover in third-party companies. Interviewee: Outsourcing (ND.01). "[outsourcing] makes it very difficult. They [i.e., the contractors] are not manageable. It's not up to us to manage how they work, their productivity. We hire contractors (...) We don't know how the work is done, by how many people or which process is executed. It is not a partnership. It is a contract." Resistance to change is the most prominent issue among respondents. As the company is public, its president and managers may change every four years, which favors some employees' skepticism. We were told that previous management initiatives were discontinued (ND.02), which caused instability among older employees, who tended to show disbelief and disinterest in using the processes. Despite the difficulties, the process group continued to improve the process (PA.05), such as i) Creation of an agile path for product development using Scrum; ii) Use of Canvas in the preliminary phase to plan projects with a smaller scope; iii) Use of kanban for task execution; iv) Gamification of the standard process to improve usability and foster dissemination of process artifacts, and v) Institutionalization of supporting tools (Mantis and Clarity). There are no SPI program goals aligned with the company's strategic plan (PA.07). There is no effective leadership to support the actions of the process improvement group (PA.08). The organizational structure is not adequate due to a lack of human resources and roles overlapping (PA.09). Lack of communication also influenced demotivation for using the process (PA.10). Interviewee: Communication (PA.10): "I think we have many problems. One of the hardest is that we have a serious problem with communication." The process meets the needs of the organization (PA.11). What hinders the use of the process is the lack of human resources to meet the demands. Process monitoring (PA.12) is not performed; no information is collected to indicate the return-on-investment (ROI). Project management was identified as the most challenging process to maintain. Interviewee: Process monitoring (PA.12): "We did [quality checklists] for a long time, but the reports we generated from non-compliance had no corrective actions because the action is not ours." Currently, the organization seeks to improve maturity in the project management process. For this, it created a group of project managers. However, the organization has no definition of whether it will undergo a new level G or F assessment in the future. 4.1.2 Organization 2 Implementation period. The organization implemented level G and later evolved to level F. In both implementations, the organization received financial assistance from the federal government. A project for SPI was defined, and people from the development team were made available. But there were no resources with dedicated time for process improvement activities. The communication of changes in the processes was in lectures and by the group of key people involved in defining the processes. The consultancy was contracted on both implementations (PA.06), and satisfaction with consultancy services was reported. Two people were hired to work in Quality Assurance Management. The main difficulties were: insufficient training, lack of resources, lack of experience in SPI, and the cultural changes that affected the oldest employees who were more resistant, for example, in the activities of configuration management. Interviewee: Resistance (PA.03). "The most difficult of all was the acceptance by people who had been here for a long time. The main thing, it was always this. People's acceptance. Unfortunately, some people did not adapt to the process, and we had to dismiss them." Maintenance period. The appraisal of organization 2 has expired. There is no intention to evolve the maturity level because managers believe that the current level meets their needs. Besides, due to the country's economic crisis (ND.03), the organization had to reduce its maintenance fees to avoid losing customers. As a result, the professionals responsible for the process quality assurance (PPQA) activities were dismissed. After the appraisal, training (PA.01) was not available for new employees. The country's economic crisis inhibits new investments in the SPI program (PA.02), reflecting on team members' motivation (PA.03) and leading to SPI abandonment. Researcher: training (PA.01): "Do they have training in the process to get in?" Interviewee: "No. Training hasn't been done lately." Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 There is no employee exclusively in charge of managing the SPI program (PA.04), and there is no strategy for introducing process improvement changes (PA.05). Concerning consulting, the organization reported satisfaction in the services provided (PA.06). Interviewee: Resources (PA.04): "Due to not pursuing further process appraisals, the quality team was dismissed. But then we reallocated the quality activities of the project to other internal people." There are no clearly defined goals (PA.07) nor a leading process group to foster continuous improvement in Organization 2 (PA.08). Although the organization is small, communication about the SPI program is flawed (PA.10); for example, there is no information available on the SPI program's benefits. Besides, Organization 2 experiences financial problems (i.e., decreased contract flow), and functions overlap due to its small size (PA.09). Interviewee: Strategic plan (PA.07): "Last year, we started putting together the organization's strategic plan, so we have the outline of it (...) But, due to time constraints, we decided not to spend too much effort as planning activities requires." The development teams partially use the process. It is not because they are considered bureaucratic (PA.11), but because there are not enough employees to execute the Quality Assurance (QA) process. Also, no Measurement Program (PA.12) exists to support process follow-up. Interviewee: Measurement (PA.12): "(...) having no financial resources, we ended up dismissed up the quality staff (composed of two employees)." 4.1.3 Organization 3 Implementation period. The organization implemented level F, evolved to level C (renewed level C once). The motivations for adopting the model were improved software processes, market, and legal need for maturity models to participate in bids. Due to the quality manager's experience in renewing level C, consultancy services (PA.06) were hired to carry out only the assessment. The organization reported satisfaction with the services provided. Maintenance period. Organization 3 intends to renew its maturity level depending on its economic recovery. The company was going through a difficult financial situation (ND.03). Therefore, the company has reduced its staff. The organization does not train its employees regularly (PA.01). However, top management supports the SPI program (PA.02) because the company participates in bids. Part of the team remains motivated to use the process because it automates activities (PA.03). Interviewee: Involvement (PA.02): "Today, I see that you can always bring improvements by sharing [experiences] with the team because I think each one knows what can improve their own process." After downsizing, Organization 3 started using open- source tools (Redmine) (PA.04). There is no process group anymore (PA.04), and the process support strategies (PA.05) are carried out by the quality manager with experience implementing the MR-MPS-SW model. Interviewee: Tools (PA.04): "So, the automation, it was fundamental to cover the lack of people." A strategic plan is aligned with the SPI program objectives (PA.07), and the communication is appropriate (PA.10). Notwithstanding, Organization 3 difficult economic situation restricts investments in an assessment to renew its maturity level. Currently, there is only one person responsible for process restructuring and monitoring (PA.09); there is no process group (PA.08). Interviewee: Structure favorable to SPI (PA.09): "In 2015, the quality team consisted of five people. In 2016, it was reduced to three people. Currently, there is only me on the quality team." Organization 3 restructured and automated the processes using a free tool (Redmine) that suits its needs (PA.11). Therefore, the processes are considered easy to maintain. Process monitoring is supported by Redmine (PA.12). Interviewee: Monitoring (PA.12): "I can't identify improvements if I don't have a minimum measurement to monitor it..." 4.1.4 Organization 4 Implementation period. The motivation for adopting the MR-MPS-SW model was to standardize organizational processes and the organization's CEO's prior knowledge, acquired in the graduate program in software engineering. Before the maturity model implementation, some teams in the organization used some Scrum practices. Thus, the consultancy helped define a process that would combine the Scrum practices with the maturity model. The main difficulties were: i) lack of support, employee involvement, ii) lack of a process group (SEPG), iii) resistance of agile teams; iv) attitude of imposition of the director (who believed in the model) and, sometimes, of the consultant; v) lack of tools; vi) lack of support from team leaders; vii) focus on the result of the assessment. Interviewee: Resistance (PA.03). "There was an area of the company that questioned the process because they worked on an already agile scheme."... "What did we do? We did a process that was a little bit tailored: some things we used a little agile, some things were a little waterfall." Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 Maintenance period. The organization does not intend to renew or evolve its maturity level. They develop software on-demand and do not participate in biddings that demand specific maturity levels. Scrum currently meets its needs. Although training (PA.01) and top management support (PA.02) were present after the assessment, the employees were unmotivated (PA.03). Due to employees who worked with Scrum on their projects, they did not accept the new process. There was also resistance from new employees to use the defined process based on the maturity model. These new employees were also resistant because they had previous experience in agile methods. Interviewee: motivation (PA.03): "so as not to follow the process, she justified: I can't. I am doing this project in Scrum, and there is no time to do anything because we have tight deadlines..." Interviewee: veiled resistance (PA.03). "…you saw that they resisted, said it was ok because the CEO was defining it, then they said it was going to be used. But it was always like this: "no, because I need to put more hours in the estimate because of the model..." The consultancy (PA.06) took into consideration the teams that worked with Scrum. However, these teams did not tell the truth to the consultant and helped define a process that would not be used after the assessment. After the assessment, the organization continued to invest in the SPI program (PA.04) and hired a process manager to make the MR-MPS-SW process compatible with Scrum. However, he had no experience with agile methods. He defined a hybrid process that was also not well accepted by the teams (PA.05). Organization 4 had a strategic plan, but it did not consider processes based on maturity models (PA.07). The SPI program did not have effective leadership in charge of process improvement (PA.08). Concerning the organizational structure, the organization has well-defined roles, which facilitate process execution (PA.09). Communication was flawed (PA.10). There was no information on SPI return on investment or benefits. The process defined in the implementation phase was abandoned shortly after the official assessment (PA.11). The lack of support from project managers and the organization's agile culture were the main reasons for the SPI initiative's failure. Project management was pointed out as the most challenging process to maintain, as the time estimated to perform activities increased due to process activities. The measurement process was abandoned after the appraisal (PA.12). Interviewee: return on investment (PA.10): "Is there information on return on investment?" Interviewee: "No. We do not have." Currently, the organization uses Scrum, Kanban, and Squads. The current CEO of the organization, with experience in agile methods, used the following strategies to manage this software process improvement initiative: i) adapt the process with agile methodologies (PA.06), aiming to meet the needs of the business; ii) training; iii) standardization of tools (Jira); iv) created the organization's Agile Manifesto (to encourage a sense of belonging); and, v) improved communication between teams. 4.1.4 Organization 5 Implementation period. Before implementing the maturity model, the organization used Extreme Programming (XP) and Kanban practices. However, the organization had a description of isolated procedures that generated the need for standardization. At the time of implementation, there were three partners, one of them actively participated in defining the processes. He participated in training on the model's processes. At that time, there was support from the owners for the SPI initiative. The main difficulties were: i) lack of human resources; ii) change of external consultancy (PA.06) (failure in the model guidelines); iii) the second consultant was located in another region of Brazil (difficulties in conducting the implementation), and iv) lack of a strategic plan. Maintenance period. Organization 5 has no interest in renewing or evolving the maturity level because the current process meets the business's needs. Besides, with the lack of external demand for certification (ND.04), there is no need to maintain an assessment using reference models because its customers do not require such evaluation. After the evaluation, the organization went through economic difficulties due to the country's financial crisis (ND.03), lost the contracts of the civil engineering sector, and started developing a predial automation software product. This affected the owners' motivation (PA.03) and support (PA.02) for SPI, who intended to implement the model's Level E. Interviewee: Country's economic crisis (ND.03). "One of our biggest customers, the civil construction company, went into crisis. So, three years ago, we lost an entire segment of civil construction…" Interviewee: Disbelief and demotivation (PA.03). "I wonder why I participated in this, but why did we invent this ...?" The organization is a micro company. Therefore, communication is easy (PA.10), and there was no need to provide training in the processes (PA.01). There is a shortage of resources and time (PA.04), and there is no SPI project Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 management (PA.05) or SPI specific goals (PA.07). It uses Redmine as a tool to support daily activities (PA.04). The dissolution of society (ND.05) was the main factor that negatively influenced SPI, because it affected the organizational structure (PA.09) and the leadership (PA.08) due to the loss of the partner who believed in the model. The process defined at the implementation time was considered bureaucratic (PA.11), being modified for Scrum practices. The current sponsor had experience with agile methods and believes that it is more effective to give the team more decision-making power than to follow processes. Project management that was considered the most bureaucratic process, was adapted with Scrum practices. In the requirements management process, user stories were used together with prototyping for requirements specification and validation. There is no measurement program for continuous improvement of the process (PA.12). Currently, the organization uses the appropriate process with agile methods because it meets the business's needs. Interviewee: the dissolution of the company (ND.05). "As the company reduced the number of employees ... because we lost a partner, we didn't have time to renew the certification." "We were in the process of making the model's E-level. But then, in this process of changing partners and getting it right, we thought it was a good idea not to do it ... We don't have to do it to get the certificate…" Interviewee: bureaucracy (PA.11). "… We fall into a planning task, and to count within our assessment, then, we had to have, for example, an action to define the communication plan. The communication plan was written once, and no one ever read it afterward... no one else used it..." 4.1.4 Organization 6 Implementation period. Organization 6 assessed level G, level F (renewed once), and Level C (renewed once) of the MR-MPS-SW model but was undecided about the second renewal of the assessment of level C due to organizational restructuring caused by the fusion of companies (ND.06). The selection of the maturity model was influenced by the sponsor, who has previous project management training. The objective was to improve the process, product quality, and market. Another strong motivator was the foreign policy to support SPI, promoted by the model's executive body (formed of a cooperative group and external financial support). The most serious difficulty was the organization's lack of experience with process improvement that resulted in a bureaucratic process (PA11). Work overload and resistance were caused (PA.03), especially for the project manager. What helped the organization achieve positive evaluation was the experience of external consultants (PA.06) and the networking between companies promoted by the cooperative group's formation. Maintenance period. After the first evaluation, senior support management continued (PA.02), made the process group (PA.08) available to make adjustments to the process, intending to reduce bureaucracy (PA.11) and increase acceptance and motivation of the organization's members (PA.03). There is a policy of continuous training (PA.01). Training needs are identified, with a technical training schedule (processes, programming language, and others) and behavioral training (motivation, integration, customer service, etc.). At the end of the training, an evaluation is made by the employees. Interviewee: training policy (PA.01). "We carry out a needs assessment at the beginning of the year with the managers."… "After the training, HR [human resources] needs to know the attendance list, the initial reaction assessment and three months later an assessment of the effectiveness of the training…" The organizational structure is adequate (PA.09), with human resources and infrastructure (PA.04) (CRM Dynamics, Pro-ject), with a strategic plan with SPI goals aligned to the business (PA.07). When the first C-Level Assessment was renewed, the organization did not use external consultancy (PA.06) because one of the process group members had experience with SPI consultancy. The process was tailored to the organization's needs. The audit of the process was automated (PA.12). The awareness of the benefits is subjective because there is no measurement of the return on investment (PA.12). SPI's management was carried out by the sponsor, who believed in process improvement and influenced top management with the process group's support (PA.08). The main support strategy used was to facilitate the use of the process through automation and reduction of bureaucracy (PA.05). However, the determining factor for the abandonment of SPI was the fusion of companies (ND.06). The fusion resulted in a clash of organizational cultures. There were changes in the business (in addition to the software factory, it started to focus on software services). There have been changes in the development process and in the way of working. The new manager of the development area encouraged discussions about the agility of organizational processes and the adhesion to the use of agile methods (ND.07), used: Scrum, squads, sprint design, and other methodologies like design thinking. Some members of the process group (PA.08) left the organization, and the process defined from the maturity model ended up abandoned. Interviewee: Fusion of companies (ND.06) - business changes. "… There was a merge with company X ... And company X brought a new portfolio. I brought an Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 infrastructure portfolio, so we have infrastructure projects now, safety nets, so we have safety nets projects, which is very different from building software…" Interviewee: Fusion of companies (ND.06) - change in the way of working. "One of the points, because of the merge, and, already advancing another point, is that it ends up that the software development process has changed a lot."… "We are reformulating our way of working."… "We are in that process like this: we certified a process, and today our process is already totally rigid. We are even looking at whether it will fit in for a reevaluation." 4.1.4 Organization 7 Implementation period. The purpose of adopting the model was the standardization of processes, product quality, market marketing, and the acquisition of public contracts (at the time, there was a requirement for evaluation using the maturity models). The SPI initiative was supported by the sponsor (PA.02), who provided hours for the project manager and some members of the organization to define the processes (PA.04), and provided model training (PA.01). People's engagement was requested (PA.03). The organization's members had no experience with SPI. What motivated the model's selection was forming a group of companies that were implementing the model in the region. Before the assessment, they used Scrum. They found the first implementation of the model more complex, with bureaucracies they were not used to (PA.11). The external consultancy was hired in both implementations of the model. However, in the second implementation, there was a conflict between the external consultant and the person responsible for implementing in the organization. It was reported that there was an exchange of consultancy because the consultancy had technical competence (PA.06) but lacked competence in soft skills. The consultancy had a very imposing posture. Interviewee: Consultancy service (PA.06). "Our ideas didn't match; he didn't accept the suggestion to change the process. "No, you have to do it this way."... "This also made it very difficult for us, especially for me, who was in charge of this company project." Maintenance period. Although the organization members have reached maturity and the processes were standardized, the sponsor has no interest in renewing the assessment (PA.02). Even meeting requirements for bids in the public sector, they did not achieve the goal defined in the strategic plan (PA.07), acquiring contracts in the public sector. Interviewee: external pressure from customers (ND.04). "…Even because concerning public projects, which was one of the ideals for us to have certification, that's not what happened..."… Researcher: "But did they ask for certification?" Interviewee: "In bidding yes." After the evaluation, there was no training available (PA.01) due to low turnover (PA.09). There were no human resources available (PA.04) to manage the SPI (PA.05), and the tools used were not adequate (PA.04). There was no group of processes (PA.08) to lead continuous improvement in processes. The members of the organization were not motivated to continue with SPI (PA.03). The process considered bureaucratic (PA.11) was adapted to the organization's needs, and they returned to using Scrum with some practices of project management and requirements management. In quality assurance management, only the quality control of the product was carried out. The other level F processes were abandoned. Interviewee: Bureaucracy (PA.11). "At level G, I felt the processes were very bureaucratic, plastered ..." The monitoring of the process stopped being done (PA.12). Therefore, there was no process institutionalization and no Information on Return on Investment (PA.10). Interviewee: Monitoring of the process (PA.12). "Today, we no longer do this audit of the process." Currently, the organization uses Scrum, and the organization members are satisfied with the reduction of bureaucracy. 4.1.5 Organization 8 Implementation period. The objective for implementing the model was to improve the process, product quality, and the acquisition of public contracts (at the time, there was a requirement for certification of models). Interviewee: objectives of SPI adoption. "We had two aspects of need. One was to improve our process, aiming for better quality."… "Except that there was also a legal need for participation in public bids." A project for SPI was defined, and people were involved in the definition of processes. Consultancy services were hired, and the sponsor was satisfied with the consultancy service (PA.06). Communication took place through engagement meetings and training (PA.10). Maintenance period. After the evaluation, no training was available (PA.01). Support from top management declined (PA.02) due to the country's economic crisis (ND.03) and the cooling of the model evaluation requirements in public bids. The organization no longer had the commercial motivation that was the requirement of Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 external customers (ND.04). These two factors affected the quality assurance process because the QA professional was not hired. Therefore, there was no monitoring of the process (PA.12). The team and the sponsor were demotivated (PA.03). The team found the process bureaucratic (PA.11). Besides, there was an overload of the product quality assurance activity, which was absorbed by the team. The sponsor thought that the documentation resulted in high costs. Interviewee: Country's economic crisis (ND.03). "I think the economic problem also helps, which is a consequence of it all."…" You see, if you don't have a crisis, you have the thriving thing."… "Then how to hire someone exclusive to the GQA? But how do you do it? The budget does not allow it. The difficulties do not allow…" Interviewee: Lack of external demand for certification (ND.04). "The bidding processes started not to charge so much because the TCU (Federal Audit Court) understands that, even, the biddings started to do as follows: if you have a certified development methodology, you present. If you don't have it, we do an audit. They kind of didn't charge. They're not charging anymore..." After the evaluation, there was no SPI management (PA.04), with the availability of resources (PA.04) and support strategies (PA.05), and no processes group (PA.08) to define continuous improvements in the process. They use Teams Foundation as a support tool (PA.04). Another factor was the turnover (PA.09) because the new employee has to learn and accept to use the process (PA.03). Interviewee: Adequate organizational structure - turnover (PA.09). "Eventually, that professional A or B who was already adhering to the process changes and then it will hurt us even more to have management." Currently, the organization no longer uses the process defined with the maturity model and adherence to agile methods (ND.07) due to the need to streamline the process and reduce documentation costs. In addition, the private market accepts Scrum well, and the public sector started to have contracts with the use of Scrum. The sponsor reported satisfaction and several benefits from simplifying the process (there is no need to keep creating evidence), reducing the conflict with the client (there is no discussion about the project scope). Interviewee: adherence to agile methods (ND.07). "We are now more with the private [sector], but with the private [sector] we can convince to use us in the agile model." 4.2 Cross-analysis This section presents the data cross-analysis of the eight organizations based on the research propositions. We used three criteria to characterize the points of analysis (Table 4): ▪ N (Not identified): the point of analysis was not identified in the organization. ▪ P (Partially identified): the point of analysis was partially identified in the organization. ▪ F (fully identified): the point of analysis was fully identified in the organization. To assess whether a proposition is confirmed, we analyzed whether the points of analysis were not identified (N) or were partially identified (P) in the organization. This means that the critical factors for maintaining SPI have been neglected. The results indicate that neglecting these factors can lead to the abandonment of the SPI program based on maturity models. To assess whether a proposition is not confirmed for the abandonment of SPI, we defined that if all points of analysis were identified (F) in the organization, it meant that the organization continues to address critical SPI maintenance factors after assessment. The following section discusses these results. Table 4. Analysis of proposition. Proposition Org. 1 Org. 2 Org. 3 Org. 4 Org. 5 Org. 6 Org. 7 Org.8 P1: There are human factors that influence the abandonment of the SPI program. PA.01. Training is offered for the qualification of the employees of the company. P N N P N F N N PA.02: There is support, commitment, and involvement of organization members. P N P P N P N N PA.03. The technical team members are motivated and willing to carry out the activities of the process. P P P N N N N N P2: There are SPI project factors that influence the abandonment of the improvement program. PA.04: Budget and resources are available for the SPI initiative. P F P F N N N N PA.05: There is a strategy to introduce changes in software processes. F N F N N N N N PA.06: Existence of an external consultancy with the ability and competence to implement a process compatible with the company's needs. - F F P F F P F P3: There are organizational factors that influence the abandonment of the improvement program. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 PA.07: Existence of a strategic plan that relates the SPI program to business goals achievement. F N F N N N F N PA.08: Leadership is available to support continuous process improvement. P N P N N N N N PA-09: There is an organizational structure favorable to the SPI program. N N N F N F N N PA-10: There are communication mechanisms for the dissemination of the SPI program N N F N F N F N P4: There are process-related factors that influence the abandonment of the improvement program. PA.11: There is a non-bureaucratic process that meets the needs of the organization. P N F F N F N N PA.12: There is a program for the measurement of continuous process improvement. N N F N N F N N 5 Discussion The research question guiding this work is: "How does the abandonment of software process improvement programs occur?" To answer this question, we conducted case studies on software organizations with either expired assessment date (organizations 1, 2, 4, 5, 7, and 8) or close to the assessment date expires (organizations 3 and 6). We identified that an organization is abandoning the improvement process when the interview participants report that all processes are no longer being used (organizations 4 and 8) or when they say that the processes are partially being used (organizations 1, 2, 3, 5, 6, and 7). We identified five pitfalls to SPI and their relation to the research questions from the data analysis. We found that organizations do not set goals to pursue continuous process improvement. There is a lack of continuity in SPI management and the sponsor's interest to continue. Even after all the effort in implementing the SPI, sponsors may not be satisfied with the results. This can lead the organization to return to its previous state or define a new way of working and improving its processes other than the maturity model. Pitfall 1 - Negligence with human factors Explanation: We found that organizations do not provide sufficient training (PA.01) (organizations 1 and 4) or have stopped providing training after assessment (organizations 2, 3, 5, 7, and 8 ). In these organizations, the lack of training negatively affected the use of the improved process because people do not use what they do not know. Training a group of people only during the SPI implementation period is not enough to ensure process understanding. The dissemination of knowledge about process improvement is complex, especially in large organizations (organizations 1 and 4), where communication can be more difficult. Top management support can influence (PA.02) the investment provisions for SPI initiatives. Organization 2 dismissed the quality team, and in Organization 3, the quality team's size was reduced to just one member. As for Organization 1 (public capital), the quality team stopped monitoring the process due to the lack of top management support. In organizations 5, 6, 7, and 8, senior management's support was perceived only during the implementation period. Regarding motivation (PA.03), we identified its partial occurrence in organizations 1, 2, and 3 because motivation depends on key people, and some people show resistance. In Organizations 4, 5, and 7 that already used agile methods before implementation, employees were resistant and unmotivated to use the new process. Organizations 6 and 8 started to adhere to agile methods (ND.07). In Organization 8, it was possible to observe the sponsor's satisfaction regarding reducing documentation costs and greater understanding with the client due to the project scope. Besides, this change in process was well accepted by its employees (especially by the younger programmers). Thus, Proposition P1 is confirmed (Table 4). Discussion: These results are consistent with the SPI literature, which reports that training is essential for disseminating knowledge (Alqadri et al., 2020) and providing awareness of the benefits of SPI (Peixoto et al., 2010). The importance of top management to be convinced about SPI's benefits for both the implementation and continuity of SPI is highlighted by Almeida et al. (2011). Resistance and lack of motivation were present in all organizational contexts. Different issues influenced them, but the lack of human resources was a common point. The resistance literature corroborates these findings when reporting that work overload discourages new work practices (Narciso et al., 2014) (Anastassiu et al., 2020). It is worth mentioning the resistance of the agile teams in organizations 4, 5, and 7. This was observed in two distinct moments: a veiled resistance by the organization members in the implementation period (due to the interest of top management in the success of the evaluation) and a more declared resistance after the evaluation. In organization 4, the teams did not use the process, even with the support of the consultancy's effort to involve these teams in discussions to define a process that would meet the organization's needs. This finding corroborates the research Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 by Albuquerque et al. (2019), which identified that teams from organizations that use agile methods have difficulties implementing and sustaining SPI based on maturity models. Pitfall 2 - Negligence with factors related to SPI projects Explanation: SPI project management is a critical success factor (Montoni et al., 2011). However, we have identified negligence in this regard. In most of the investigated organizations, it was possible to observe that in the implementation period there was a definition of a project with availability of dedicated resources (PA.04). However, after the evaluation there was no continuity in the management of the SPI project. In other organizations (for example, 2 and 7), lack of management occurs even during the implementation period. The lack of a dedicated resource (PA.04) to manage SPI negatively affects the continuous improvement of the process and the taking of actions to promote people's motivation, that is, the definition of SPI support strategies (PA. 05). Only Organization 1 has a process group (PA.04) that continues to take actions (PA.05) to promote SPI. However, it is difficult for a process group to keep the SPI program running without senior management support (PA.02). In organizations 3 and 6, processes were automated to increase compliance (PA.05). Regarding the analysis of this proposition, our data were not conclusive to confirm this proposition because the analysis point regarding the consultancy (PA.06) was not possible to evaluate in all organizations. For example, organization 1 did not hire consultancy services. Thus, proposition P2 is partially confirmed (Table 4). Discussion: According to SPI literature (Montoni et al., 2011) (Coleman et al., 2008) (Peixoto et al., 2010) (Almeida et al., 2011), SPI initiatives are affected by the lack of human resources, resulting in work overload and, therefore, in the prioritization of activities related to the product. According to Sulayman et al. (2012), the SPI team needs to have the workforce available to define the processes, train the team members on these processes and supervise. For this reason, having a full-time person for coordination activities is essential for the success of the SPI initiative (Guerrero et al., 2004). Pitfall 3 - Negligence with organizational factors Explanation: There are no clearly defined goals (PA.07) or effective leadership (PA.08) of top management and project managers that foster continuous improvement. Besides, there is role overlapping (PA.09), and communication is flawed (PA.10). Only Organization 4 had no role overlapping. However, agile culture hinders the acceptance of the new processes. This difficulty also occurred in organizations 5 and 7, which already used agile methodologies before implementation. We identified two new results: Dissolution of the company (ND.05) and fusion of companies (ND.06) that affected the organizational structure, resulting in SPI abandonment. In organization 5, the Dissolution of society (ND.05) negatively affected the SPI initiative because it lost its leadership. That is, it lost the person who believed in the model. Thus, the organization returned to agile methods because the remaining partners believe in agile methods' value. In organization 6, the fusion of companies impacted SPI's abandonment because there was a restructuring of organizational processes. In this restructuring, the new development manager with agile methods' experience defined a new way of working with senior management support. Thus, Proposition P3 is confirmed (Table 4). Discussion: The importance of considering organizational culture in SPI initiatives was reported in the research (Alqadri et al. 2020) Shih et al. (2010). Shih et al. (2010) emphasized that SEPG (Software Engineering Process Group) leaders should consider culture when a new SPI approach is implemented because it may be incompatible with the existing culture. In organizations 4, 5, and 7 with organizational cultures used to working with agile methodologies, it was challenging to continue SPI with maturity models. We identified that groups, such as the process group and the quality assurance group, made the most effective support and leadership to sustain SPI. Our results are consistent with the research by Uskarci and Demirörs (2017). Regarding the new findings, it was possible to observe the influence that the organizational structure has on SPI initiatives and how they are related to knowledge and previous experience in process methodologies and decision making. In organizations 4, 5 and 6, the choice was made to use agile methods due to the organization's previous experience of managers with decision-making power. Pitfall 4 - Negligence with process factors Explanation: Regarding the existence of a non- bureaucratic process (PA.11), we found that all organizations adjusted and simplified their processes after the official assessment. In organizations 1, 2, 6, and 7, the process is partially used (quality assurance and measurement are not performed). Organizations 4 and 8, which have an agile culture, abandoned the processes thoroughly. Notably, only Organization 3 (which participates in bidding processes) continued to use and monitor the processes (PA.12). However, it had not renewed the maturity level because they experienced financial struggles by the interview time. We found that some organizations abandoned SPI with maturity models due to adherence to agile methodologies (ND.07), as was the case with organizations 6 and 8. These are organizations that started using agile methods after the evaluation. Its sponsors reported satisfaction with using these methodologies due to the reduction of bureaucracy and documentation costs. Thus, Proposition P4 is confirmed as can be seen in Table 4. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 Discussion: The results showed that abandoning the SPI program does not mean not using the organizational processes at all. Organizations 1, 2, and 3 have adapted and simplified their processes to meet their new business needs. These results align with the SPI literature, which reports that processes tend to be simplified, stabilizing in a minimum process (Coleman et al., 2008). Organizations 4, 5, 6, 7, and 8 have been looking for other ways to mature the process using agile methods (Fontana et al., 2015). It is worth mentioning that it is possible to implement an SPI initiative with agile methodologies and maturity models. However, in the context of this research, only organization 4 tried to make this tailoring but was unsuccessful due to the boycott of agile teams. Pitfall 5 - Negligence with external factors Explanation: We identified external factors that impact the support of top management. We identified the negative impact of outsourcing (ND.01) IT projects on organization A (a large public company). Project managers reported difficulty in applying their processes to outsourced organizations. The main reason was the high turnover that made learning difficult and hindered the use of the processes. The country's economic crisis (ND. 03) has restricted investments in resources for SPI. Also, we found that regular changes in the state government (ND.02) demotivates process managers from adhering to the changes made by top management because the company's board can change every four years and, therefore, potentially change the internal software process quality policies. The lack of external pressure from customers (ND.04) is another factor that discouraged some organizations that had the commercial motivation to adopt SPI with maturity models, that is, the interest in participating in public biddings. However, currently in the country, this requirement has not been made by all public bodies. Organizations working in the private sector have reported no requirements to use an officially evaluated process. Discussion: Unlike the literature, our study identified new findings negatively influencing SPI, called external factors. Outsourcing (ND.01) impacted the lack of use of the improvement process due to the lack of standardization of outsourced contracts. This indicates that it is vital for the organization's top management to define procedures for managing third-party contracts. Regarding the regular changes in the state government (ND.02), the results show that consistency in quality policies is necessary. The frequent change in the use of software process methodologies, or the definition of work procedures, may demotivate organization members at any organizational level. It is quite possible that this lack of managerial constancy may demotivate members in private organizations as well. Here, it is a point worth investigating. The country's economic crisis (ND.03) has been affecting organizations' economic instability. These organizations have a reactive action to decrease their resources, prioritizing the resources that develop the software and dismissing the quality team. Finally, the lack of external pressure from the client (ND.04) indicates that the organizations that adopted the SPI for purely commercial reasons and not improving processes themselves tend to be frustrated with the results because the public sector has changed its way of acquiring software development services. Thus, we formulated a new proposition: P5. There are external factors that influence the abandonment of the improvement program. 6 Limitations and Threats to Validity To evaluate the research quality and research validity, we used the guidelines defined by Yin (2017) and Runeson et al. (2012) regarding quality criteria for empirical research. Regarding construct validity, the propositions are based on the research carried out by Albuquerque et al. (2018). Propositions and analysis points were validated in a workshop held with experienced professionals in SPI programs. Regarding internal validity, Grounded Theory procedures were followed: the propositions were investigated using only the data collected from the interviews. The first author analyzed the interviews and built the networks. The other authors (professionals with experience in maturity models implementation and assessment) reviewed and analyzed quotes, codes, and categories. Regarding external validity, we interviewed participants from eight different software organizations. We included organizations of various sizes, locations, and businesses. Three organizations do not participate in biddings, and only one is a public company. Some organizations only provided one participant for the interview (due to high turnover). Still, we were careful to select those who effectively participated since the maturity model implementation. As expected in in-depth qualitative research, the results cannot be broadly generalized (Eisenhardt, 1989) but present relevant evidence on how abandonment occurs after valid SPI appraisals. Nonetheless, we plan to replicate the research in more organizations. Finally, to ensure research reliability, all the research protocol and data analysis steps were defined and followed. 7 Conclusion This study aimed to understand how abandonment occurs in SPI programs after successful assessments based on maturity models. Results from four organizations (1, 2, 3, and 4) were published in Albuquerque et al. (2020), who indicated that abandonment occurs when there is negligence to factors internal to the organization (Human, Organizational, SPI Project and Processes) and factors external to the organization (Outsourcing - ND.01, Political Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 change - ND.02 and Economic crisis of the country - ND.03). In this paper, results from four more organizations (5, 6, 7, and 8) were presented. Concerning internal factors, they all corroborated our previous research (Albuquerque et al., 2020). However, new findings were identified: two Organizational factors (Dissolution of the company - ND.05 and Merger of companies - ND.06) and a Process factor (Adherence to agile methodologies - ND.07). Concerning external factors, this research confirmed the negative influence of the country's economic crisis on SPI and identified a new external factor (Lack of external demand for certification - ND.04). Another point that draws attention is that some organizations carried out management activities during the SPI project until the official assessment. After that, some of them neglect the proper management of the SPI project. Moreover, other organizations neglect management activities since the beginning of the SPI project. Considering that the literature and our expirence state that adequate management is a critical success factor to the SPI project, it is not surprising that such organizations will fail to continue the SPI activities carried out so far. As a contribution, we highlight the practical applicability of our results for the software industry. Industry professionals can use this study's results to learn about their initiatives to avoid pitfalls that can lead to abandoning SPI. For example, before starting an SPI initiative, evaluate the organization's business, and assess whether it is the best time to invest in process improvement. Evaluate if the organizational structure is appropriate if there is a flow of ongoing projects to avoid the investment restriction with training; and reduce the team, such as the quality team. Before starting an SPI initiative, know the improvement model that will be implemented, and be aware that the results come in the long term. It is also essential to involve the development team in selecting the process improvement model and the process definition to avoid resistance. The consultancy will only help define a valuable process for the organization, but the development team's commitment will lead to SPI success. The technical skill of the consultancy is useless without the spontaneous participation of the team members. Effectively combining agile methods and maturity models requires experienced consultants to overcome this integration's natural barriers. A balanced process can combine agile methods and model requirements in a sustainable path As future work, we are starting to replicate this study in other software organizations that use maturity models (MPS- SW and CMMI), considering different sizes, maturity levels, companies capital, and organizational contexts. Our goal is to deepen our understanding of the movement organizations makes after the official appraisal. Acknowledgments We thank the financial support provided by the Araucária Foundation (FA). Agreement number: 001/2017. We also thank UNIRIO for its financial support (Edital PPQ-UNIRIO 2019 and 2020). References Albuquerque, R., Fontana, R.M., Malucelli, A., Reinehr, S. (2019). Agile Methods and Maturity Models Assessments: What's Next? In: Proceedings of the Systems, Software and Services Process Improvement (EUROSPI), Edinburgh, Scotland, pp 619-630. Albuquerque, R., Malucelli, A., Reinehr, S. (2018). Software Process Improvement Programs: What happens after official appraisal. In: Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE), San Francisco, USA. Albuquerque, R., Santos, G., Malucelli, A., Reinehr, S. (2020). Abandonment of a Software Process Improvement Program: Insights from Case Studies. In: Proceedings of the Brazilian Symposium on Software Quality (SBQS), Maranhão, Brazil. Almeida, C.D.A., Albuquerque, A.B., Macedo, T. C. (2011). Analysis of the continuity of software processes execution in software organizations assessed in MPS.BR using Grounded Theor. In: Proceedings of the International Conference on Software Engineering and Knowledge Engineering (SEKE), Miami, Florida, USA. Alqadri, Y., Budiardjo, E. K., Ferdinansyah, A., Rokhman, M. F. (2020) The CMMI-Dev Implementation Factors for Software Quality Improvement: A Case of XYZ Corporation. In: Proceeedings of the 2nd Asia Pacific Information Technology Conference (APIT), pp.34-40. Anastassiu,M., Santos, G. (2020). Resistance to Change in Software Process Improvement - An Investigation of Causes, Effects and Conducts. In: Proceedings of the Brazylian Symposium on Software Quality (SBQS), Maranhão, Brazil. Canedo, E. D., Santos, G. A. (2019). Factors Affecting Software Development Productivity: An empirical study. In: Proceedings of the XXXIII Brazilian Symposium on Software Engineering (SBES), September in Brazil. p.3017-316. CMMI INSTITUTE (2018). CMMI for Development v2.0. Available at: https://cmmiinstitute.com/products/cmmi/cmmi-v2- products. CMMI INSTITUTE (2019). Radix: Delivers Results with CMMI and Behavioral Driven Development in Agile Environment. Submitted by: CMMI Institute. Published: 25 July, 2019. Coleman, G., O'Connor, R. (2008). Investigating software process in practice: A grounded theory perspective. Journal of Systems and Software, v.81, issue 5, p.772-784. Software process improvement programs: What are the pitfalls that lead to abandonment? Albuquerque et al. 2021 Eisenhardt, K. (1989). Building Theories from Case Study Research. Academy of Management Review, v. 14, issue 4, pp. 532-550. Fontana, R.M., Meyer, Jr. V., Reinehr, S., Malucelli, A. (2015). Progressive outcomes: A framework for maturing in agile software development. Journal of Systems and Software, v. 102, pp. 88-108. Guerrero, F., Eletrovic, Y. (2004). Adopting the SW-CMM in a small IT organization, IEEE Software, v.21, issue 4, July-Aug. 2004, pp.29-35. ISO/IEC (2015). ISO/IEC 33020:2015: Information Technology - Process Assessment – Process measurement framework for assessment of process capability, Geneve: ISO. ISO/IEC (2017). ISO/IEC/IEEE 12207:2017 Systems and software engineering. Software life cycle processes. Kalinowski., M., Weber, K., Franco, N., Zanetti, D., Santos, G. (2014). Results of 10 Years of Software Process Improvement in Brazil Based on the MPS-SW Model. In Quality of Information and Communications Technology (QUATIC) in Portugal, p. 28-37. Montoni, M.A., Rocha., A. R. C. (2011). Using grounded theory to acquire knowledge about critical success factors for conducting software process improvement implementation initiatives. International Journal of Knowledge Management, v.7, issue 3 (jul 2011), pp. 43– 60. DOI: 10.4018/jkm.2011070104. Nalepa, G., Fontana, R.M., Reinehr, S., Malucelli, A. (2019). Using Agile Approaches to Drive Software Process Improvement Initiatives. In: Proceedings of the Systems, Software and Services Process Improvement (EUROSPI), Edinburgh, Scotland, pp 495-506. Narciso, H; Allison, I. (2014). Overcoming structural resistance in SPI with Change Management. In: Proceedings of the International Conference on the Quality of Information and Communications Technology (QUATIC), p.8-17. O'Connor, R (2012). Using Grounded Theory Coding Mechanisms to Analyze Case Study and Focus Group Data in the Context of Software Process Research. Published in the United States of America by Information Science Reference (an imprint of IGI Global), 2012. Cap.13, p.256- 270. DOI: 10.4018/978-1-4666-0179-6.ch013. Peixoto, D.C.C., Batista, V. A., Resende, R.F., Isaías, C. (2010). How to Welcome Software Process Improvement and Avoid Resistance to Change. In: Proceedings of the International Conference on Software Process (ICSP), Alemanha, p.138-149. Reinehr, S., Pessôa, M S. P., Burnett, R.C. (2008). Software product lines in the financial sector in Brazil. In: Proceedings of the XXVIII National Congress on Production Engineering (ENEGEP). Rio de Janeiro, Brazil. Runeson, P., Host, M., Rainer, A., Regnell, B. (2012) . Case Study Research in Software Engineering: Guidelines and Examples.. March 2012 256 pages. Shin, C.C., Huang, S.J. (2010). Exploring the relationship between organizational culture and software process improvement deployment, In Information & Management, v.47, p.271–281. Society for the Promotion of Brazilian Software Excellence – SOFTEX (2020). MPS General Guide to Software. http://www.softex.br/mpsbr. Strauss, A., Corbin, J. (1998). Basics of Qualitative Research, 2ª ed.: Sage Publications, Thousand Oaks, London New Delhi, 1998, 312p. Sulayman, M., Urquhart, C., Mendes, E., Seidel, S. (2012). Software process improvement success factors for small and medium Web companies: A qualitative study, In Information and Software Technology v.54, p.479–500, 2012. Uskarci, A., Demirörs, O. (2017). Do staged maturity models result in organization-wide continuous process improvement? Insight from employees. In Computer Standards & Interfaces, v.52 p.25–40. Yin, R. (2017). Case Study Research: Design and Methods (Applied Social Research Methods), 6th edn. Los Angeles: Sage Publications. Software process improvement programs: What are the pitfalls that lead to abandonment? 1 Introduction 2 Related works 3 Research Method 3.1 Context 3.2 Data collection 3.3 Data analysis 4 Results 4.1 Analysis of individual cases 4.1.1 Organization 1 4.1.2 Organization 2 4.1.3 Organization 3 4.1.4 Organization 4 4.1.4 Organization 5 4.1.4 Organization 6 4.1.4 Organization 7 4.1.5 Organization 8 4.2 Cross-analysis 5 Discussion 6 Limitations and Threats to Validity 7 Conclusion Acknowledgments References