Journal of Software Engineering Research and Development, 2022, 10:1, doi: 10.5753/jserd.2021.1973  This work is licensed under a Creative Commons Attribution 4.0 International License.. TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Eliezer Dutra  [ UNIRIO and CEFET/RJ | eliezer.goncalves@cefet-rj.br ] Patrícia Lima  [ UNIRIO | patricia.lima@edu.unirio.br ] Cristina Cerdeiral  [ Univeris | cerdeiral@gmail.com ] Bruna Diirr  [ UNIRIO | bruna.diirr@uniriotec.br ] Gleison Santos  [ UNIRIO | gleison.santos@uniriotec.br ] Abstract Background: Measuring the organizational climate of agile teams is a challenge for organizations, mainly because of the shortages of specific instruments to agile methodologies. On the other hand, finding companies willing to participate in the preliminary validation of an instrument is a challenge for researchers of the organizational climate. The preliminary validation allows identifying problems and improvements in the instrument. Objective: We present the preliminary evaluation of TACT. TACT is an instrument to assess the organizational climate of agile teams. Its initial version comprises the Communication, Collaboration, Leadership, Autonomy, Decision-Making, and Client Involvement dimensions. Method: We planned and executed a case study considering three development teams. We evaluated TACT using open-ended questions, quantitative methods, and TAM dimensions of Intention to Use, Perceived Usefulness, and Output Quality. Results: TACT allowed to classify the organizational climate of the teams for the Communication, Collaboration, Leadership, Autonomy, Decision-Making, and Client Involvement dimensions. Some items were assessed negatively or neutrally, which represent points of attention. TACT captured the lack of agile ceremonies, the difficulty of the product owner in planning iterations, and the distance in leadership. In addition, TACT dimensions presented high levels of reliability. Conclusions: TACT captured the organizational climate of the teams adequately. The team leaders reported intention of future use. The items that compose TACT can be used by researchers investigating the influence of human factors in agile teams and practitioners who need to designorganizationalclimateassessmentsofagile teams. Byusinganinstrumentadaptedtoassesstheorganizational climate of agile teams, an organization can better identify issues and improvement actions aligned with agile values, principles, and practices. Keywords: Organizational climate, Agile software development, Human factor influence 1 Introduction Several factors can influence the organizational climate of agile software development teams, such as trust, openness, respect, team engagement, a culture of action and change, in- novation, leadership, communication, personality, software quality, performance, support from management and the availability of resources for the project (Acuña et al., 2008; Soomro et al., 2016; Grobelna and Stefan, 2019; Serrador et al., 2018; Vishnubhotla et al., 2020). Curtis et al. (2009) propose that organizations should periodically identify each person’s opinion on their working conditions. The authors recommend the organizational climate survey to learn and understand the factors influencing teams, their activities, and, consequently, the software’s quality (Curtis et al., 2009). The instrument used in the assessment of the organiza- tional climate must consider the most critical factors in the domain, as the organizational climate is evaluated through behavior, attitudes, feelings, policies, practices, and proce- dures that characterize life in the organization (Lenberg et al., 2015; Schneider et al., 2014). Vishnubhotla et al. (2020) point out the need for further studies to investigate the in- fluence of human factors on the organizational climate of agile teams. Both academia and industry suggest that collab- oration, communication, autonomy, decision-making, client involvement and leadership are critical human factors that influence agile software development projects (Chagas et al., 2015; Dybå and Dingsøyr, 2008). To assess the organizational climate of agile teams, organi- zations should select the organizational climate instruments that measure the desired factors. Many organizations may find it difficult to select instruments for copyright reasons. Hiring a specialized consulting company can aid this process. However, Dutra et al. (2012) report that many consulting companies do not disclose details of how the instrument was designed, its reliability, nor the statistical procedures adopted to its validation. Several studies have investigated the impact of human fac- tors in agile projects (Chagas et al., 2015; Vishnubhotla et al., 2018), including surveys with members of agile teams (Gro- belna and Stefan, 2019). However, the literature review we conducted did not identify studies that report the design of scales, models, or questionnaires specific to assess the orga- nizational climate of agile teams. Some studies use generic scales/questionnaires that can be used in different business domains (Acuña et al., 2008; Vishnubhotla et al., 2020). Other studies only present factors that exert some influence on the organizational climate of agile teams (Serrador et al., 2018; Soomro et al., 2016). In previous work, Dutra et al. (2020) presented the ini- tial version of TACT: “an insTrument to Assess the or- ganizational ClimaTe of agile teams”. TACT was devised https://orcid.org/0000-0002-9000-8369 mailto:eliezer.goncalves@cefet-rj.br https://orcid.org/0000-0002-2637-011X mailto:patricia.lima@edu.unirio.br https://orcid.org/0000-0002-3443-2202 mailto:cerdeiral@gmail.com https://orcid.org/0000-0002-1197-9133 mailto:bruna.diirr@uniriotec.br https://orcid.org/0000-0003-0279-0440 mailto:gleison.santos@uniriotec.br TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 and validated preliminary for the Communication, Collabora- tion, and Leadership dimensions. The instrument dimensions showed high reliability. In the current work, we extended the initial study by adding the Client Involvement, Auton- omy, and Decision-Making dimensions, creating new items to measure the organizational climate of the teams consid- ered in the previous study, and expanding the users of TACT to include a third team. Moreover, we increased the litera- ture background to show the constructs (Delgado-Rico et al., 2012) considered to guide the creation of TACT items, and we used Factor Analysis to identify the most influential items for each dimension considered in the case study. This study aims to evaluate TACT preliminarily for the Communication, Collaboration, Leadership, Auton- omy, Decision-Making, and Client Involvement dimensions. TACT was built considering the main human factors that in- fluence agile teams. Two specialists confirmed the validation of the TACT items for agility. The data collection procedures used in the case study showed that TACT evaluated the orga- nizational climate correctly for the three teams. The quanti- tative analysis indicated the most influential items by each dimension in the case study. TACT items showed high fac- tor loading. TACT showed excellent psychometric indices, for example, high correlation inter items in the Spearman correlations (ρ) and high alfa Cronbach value (> 0.8). Prac- titioners can use TACT items in their organizational climate assessment. Researchers can explore new evidence of relia- bility and validity of the TACT dimensions. The paper is organized as follows: Section 2 discusses the organizational climate in agile teams; Section 3 presents the design of TACT; Section 4 deals with the study planning; Section 5 presents the results; In Section 6, we discuss the results; Section 7 addresses the study limitations and threats to validity; finally, Section 8 shows our final considerations. 2 Background 2.1 Specific Characteristics for the Formation of the Organizational Climate of Agile Teams The organizational climate is the meaning that employees at- tribute to the policies, practices, and procedures they expe- rience, besides the behaviors they observe being rewarded, supported, and expected (Schneider et al., 2014). As such, members of agile teams expect the values, practices, adopted procedures, and, even, the behavior of those involved to re- flect the values, principles, and practices of the “agile philos- ophy” (Hohl et al., 2018; Beck et al., 2001). Agile methods differ from traditional development meth- ods in several aspects (Dybå and Dingsøyr, 2008; PMI and Agile Alliance, 2017). Leadership, collaboration, communi- cation, autonomy, decision-making, and client involvement are examples of factors that demand different behaviors among those involved, as they impact the adoption and use of agile methods (Dybå and Dingsøyr, 2008; Chagas et al., 2015; Noll et al., 2017; Jia et al., 2016). Schneider et al. (2014) claim that leadership is a crucial point in the formation of the climate in organizations. In ag- ile development, the leadership is based on the role of the Servant Leader (PMI and Agile Alliance, 2017). PMI and Ag- ile Alliance (2017) argue that servant leadership is the prac- tice of leading by service, focusing on the team members’ comprehension, development as well as meeting their needs to enable them to perform at their best. Dybå and Dingsøyr (2008) argue that, in traditional methodologies, the manage- ment style is based on command and control with highly bu- reaucratic and formalized organizational structures, while in agile methodologies, the management style must be collabo- rative and the structure of the organization is organic (Dybå and Dingsøyr, 2008). Chagas (2015) reports that collaboration in agile method- ologies takes place between team members and the customer. In agile methodologies, the project is divided into small cy- cles, called iterations, which are planned and specified ac- cording to the client and based on the team’s development capacity (PMI and Agile Alliance, 2017). This negotiation is based on the communication and collaboration the team has while executing the development tasks. A process of com- munication and collaboration between members of the agile team in the iteration planning and the development tasks exe- cution positively impacts the project’s success (Chagas et al., 2015). Unlike traditional approaches, in agile methodologies, the team has the autonomy to create and change the responsibil- ity for performing the tasks (Karhatsu et al., 2010; Chagas, 2015; PMI and Agile Alliance, 2017; Noll et al., 2017). Jia et al. (2016) argue that the decision-making behavior of each individual will influence the behaviors of other teammates and the project outcome. For example, each member makes a decision about effort estimation and gives user story points under these conditions; different individual decision-making behaviors will generate different results, which are pertinent to the success or failure of the project. (Jia et al., 2016). Dutra and Santos (2020) investigated difficulties associ- ated with organizational climate assessments. The authors identified pitfalls in (i) non-assessment of behaviors and fac- tors specific to the development of an organizational cli- mate in agile teams, and (ii) not explicitly considering ag- ile roles and other organizational structure management func- tions. The authors argue that the items of assessment instru- ments should be detailed enough to allow respondents to think about the organizational culture and better characterize the agile behaviors depicted (Dutra and Santos, 2020). 2.2 Organizational climate in agile teams There are several studies on organizational climate in soft- ware development teams (Soomro et al., 2016). However, many of these studies do not report characteristics of the soft- ware development process considered in the evaluated teams. In addition, the studies measured the climate using generic in- struments used in different business domains, without consid- ering values, principles, or specific practices of development teams. Our literature review identified three studies (Acuña et al., 2008; Grobelna and Stefan, 2019; Vishnubhotla et al., 2020) that investigated the organizational climate of agile teams by survey climate instruments. TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Acuña et al. (2008) investigated whether the climate of software developers teams has any relationship with the qual- ityof thesoftwareproduct.TheauthorsusedtheTCI©(Team Climate Inventory) instrument (Anderson and West, 1998) to assess the climate. The experimental study was carried out with 105 students allocated in 35 teams. All teams used an adaptation of the eXtreme Programming method (XP) to de- velop the same software. The authors found that the climatic preferences of the team’s vision and their perception of par- ticipatory security were significantly correlated to better soft- ware. According to the authors, it is important to track the organizational climate of teams as one of many indicators of the quality of the software to be delivered. Grobelna and Stefan (2019) investigated how the organi- zational climate factors (e.g., Leadership Style, Autonomy, Rewarding, and Communication) in agile software develop- ment teams affected the regularity of work speed and the teams’ efficiency. The authors prepared a questionnaire to measure the organizational climate, but the items created were not disclosed. The results confirmed that the desired organizational climate was based primarily on a positive re- lationship with the leader and other coworkers, commitment to work, and challenges at work. The authors argue that there are elements that point out that the more the team’s organiza- tional climate is characterized by the team’s preferences, the greater the regularity of the work speed of this team is, and thus the team is more efficient (Grobelna and Stefan, 2019). Vishnubhotla et al. (2020) investigated the association be- tween personality traits and the climate in agile software de- velopment teams. The study was implemented with 43 mem- bers in eight agile teams. The authors used the TCI© instru- ment (Anderson and West, 1998) to assess the climate for each dimension (vision, participatory security, support for innovation, and task orientation). The study identified a sta- tistically significant positive correlation between personality (considering the trait openness to experience) and the climate dimension (support for innovation). They concluded that the results of the regression analysis suggest that more data may be needed, and there are other human factors in addition to personality traits that should also be investigated in relation to the climate of agile teams. In summary, the TCI© instrument is grounded in a theoret- ical model to measure vision, participatory security, support for innovation, and task orientation dimensions (Anderson and West, 1998). TCI© was used in Acuña et al. (2008) and Vishnubhotla et al. (2020) to measure factors that influence the capability of innovation of software development teams. The TCI© dimensions do not measure the dimensions pro- posed on TACT. The questionnaire items elaborated by Gro- belna and Stefan (2019) were not published. Regarding the use of questionnaires or generic scales to assess the organiza- tional climate in agile teams, Dutra and Santos (2020) claim that the use of assessment instruments that do not consider ag- ile values, principles, practices, and roles in a proper context may create difficulties for the analysis of possible causes of problems and the execution of corrective actions within orga- nizational climate management. Therefore, there is a need for specific instruments to measure the organizational climate of agile teams in the communication, collaboration, leadership, autonomy, decision-making, and client involvement dimen- Figure 1. Main steps used to build TACT and to execute the case study sions. 3 TACT Overview In this section, we present the conception of the insTru- ment to Assess the organizational ClimaTe of agile teams (TACT). Instruments for organizational climate assessments measure behaviors, attitudes, or preferences (Anderson and West, 1998; Patterson et al., 2005). As such, TACT con- ception and evaluation are based on psychometry concepts (Dima, 2018; Patterson et al., 2005; Graziotin et al., 2020). TACT design followed specific procedures suggested for elaborating and validating climate scales and other question- naires in general (Graziotin et al., 2020; Anderson and West, 1998; Bandura, 2006; Dybå, 2000; González-Romá et al., 2009; Recker, 2013; Shull et al., 2008). Figure 1 shows the steps followed to define TACT and to execute the case study used to evaluate it. The steps involving the definition of constructs, items design, evaluation by spe- cialists, and pretesting are described in the next subsections. The activities used for data collection from the case study, such as the interview with the process coordinator, documen- tation analysis, a survey using TACT, leaders interview, and TAM evaluation, are in Section 4.3. The quantitative analysis from the case study is shown in Section 5.3. 3.1 Conceptual definition of the construct The first step to define the construct is a literature review (Spector, 1992). The researchers should carefully read the literature about the construct, paying attention to specific de- tails of exactly what the construct has been described (Spec- tor, 1992). In the delineation of a construct, it is helpful to base the conceptual and scale development effort on work that already exists. For each TACT dimension, we identified (i) conceptual definitions to show a general description of the construct measured, and (ii) operational definitions to under- stand how the construct can be assessed (Delgado-Rico et al., 2012; Spector, 1992). An operational definition is a descrip- tion of something in terms of the operations (procedures, ac- tions, or processes) by which it could be observed and mea- sured (VandenBos, 2017). The constructs are presented in Appendix A.1. To start step 1, we identified systematic literature reviews and other relevant sources to provide (i) theoretical and oper- ational definitions for the investigated constructs (i.e., com- TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 munication, collaboration, leadership, autonomy, decision- making, and client involvement), (ii) human factors and their influences on agile teams, and (iii) factors, models, scales, questionnaires, and items for assessing climate of software development teams. We have identified some systematic lit- erature reviews about human factors that impact agile soft- ware development (Dybå and Dingsøyr, 2008; Franca et al., 2011; Chagas et al., 2015; Vishnubhotla et al., 2018; Dutra et al., 2021). Soomro et al. (2016) paper was considered for having identified studies, instruments, and factors used to as- sess the organizational climate of development teams. PMI and Agile Alliance (2017) and Miller (2020) were used to standardize names of roles, practices, and artifacts consid- ered in agile development. We used the most influential hu- man factors related to Agile Software Development teams (Chagas et al., 2015) to select the TACT dimensions investi- gated in this study. The Agile Manifesto (Beck et al., 2001) was also used in this step. The identified literature was used (i) to make the concep- tual and operational definition of constructs (Delgado-Rico et al., 2012; Spector, 1992) and (ii) to capture examples of behaviors, attitudes, climate instruments, and practices and their influences. For example, a) Dybå and Dingsøyr (2008) showed that “the planning game activity was found to have a positive effect on collaboration within the company”, b) Karhatsu et al. (2010) reported that “communication and col- laboration are at the heart of agile software development. As the Agile Manifesto states, individuals and interactions over processes and tools and customer collaboration over contract negotiation. One aspect in communication and col- laboration is customer cooperation”, and c) through Soomro et al. (2016), we identified some items (Açıkgöz et al., 2014) that could be adapted to measure the collaboration. 3.2 Design/adaptation/selection of items Step 2 aims to propose items that will be used to assess each dimension, adapted to the population’s culture. Thus, the con- structs (Appendix A.1) identified in Step 1, the identified systematic reviews, and other relevant literature were con- sidered. Some items or questionnaires and examples of behav- iors identified in the previous step must be adapted to ag- ile roles, practices, or artifacts. PMI and Agile Alliance (2017) and Miller (2020) were considered a reference to identify the main roles and essential activities in agile soft- ware development projects. After reading the selected works, we started creating TACT. For each considered dimension, namely Communication, Collaboration, Leadership, Auton- omy, Decision-making, and Client involvement, evaluation items were selected, adapted, or created. Some items from scales without any copyright were se- lected and translated to Portuguese, e.g., “IT13. Team mem- bers work together as a whole” used in Açikgöz (2017) to as- sess collaboration between software development team mem- bers. In other cases, only the role of the person exercising the action was altered. For example, the original item “My direct supervisor listened to my ideas and concerns”, proposed in Sharma and Gupta (2012), was changed to item “IT20. The team facilitator listens to my ideas and concerns”. New items were also proposed to assess the organizational climate specific to agile teams. For this purpose, critical fac- tors and/or items were selected, and the descriptions were adapted to the roles and activities performed by agile teams. For example, to assess the Communication dimension, we defined the item “The team and the product owner always reach consensus on the priority of the user stories by negoti- ating which bug to fix or functionality to add”. This item was based on the team climate factor described in Nianfang Ji and Jie Wang (2012) “Supervisors and staff communication and agreement their tasks, including what to do, to what degree, and how to do?” and the description presented by Chagas (2015) for the communication factor “Frequent communica- tion can be used to prioritize features, set focus on bug-fixing or include more functionality”. On completion of Step 2, 49 items had been established, with 9 items to measure Communication, 8 items for Collab- oration, 10 items for Leadership, 7 items for Autonomy, 8 items for Decision-Making, and 7 items for Client Involve- ment dimension. The items included in the TACT initial ver- sion are shown in Appendix A.2. TACT also comprises a dashboard, which is shown in Section 5. 3.3 Evaluation by specialists At the beginning of Step 3, the TACT items were analyzed by two specialists in agile software development. For each item, two questions were considered “Can it be inferred that the presented item represents a behavior related to agile soft- ware development teams?” and “Do you suggest any adap- tation to the item description?”. The first specialist has 10 years of experience in using such methods and 5 years as a consultant focused on the agile transformation of organiza- tions and teams. The second specialist is a process coordi- nator at a huge company. She has 14 years of experience in software process improvement and 4 years as responsible for defining and monitoring changes in agile processes. Every TACT item was considered related to agile soft- ware development teams. Two researchers, co-authors of this work, discussed all comments and suggestions made by the specialists. After that, some adaptations in item descriptions were made. For example, in IT08, the proposed description “The team and the product owner always agree (...)” was altered to “The team and product owner always reach con- sensus (...)”. 3.4 Pretesting Google Sheets was used as a tool to develop TACT. It mostly contains the form for conducting the climate survey and a dashboard with the results of the frequency by item and di- mension (Figure 2). The items proposed in Appendix A.2 are measured using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree and 5 = strongly agree). In TACT, the organizational climate of the team is classified as positive (values 5 and 4), neutral (value 3), or negative (values 2 and 1). To begin Step 4, a pretesting was performed with 3 de- velopers to identify possible problems of interpretation for TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 the TACT items and layout. In the end, the developers re- ported no difficulties in answering the survey. The authors implemented a layout suggestion presented in this step. To continue the preliminary assessment of TACT, a case study (Yin, 2013) was performed and is described in the next sec- tion. 4 Case Study Planning and Execution Runeson and Höst (2009) claim that case studies in software engineering aim to investigate a contemporary phenomenon in a real context for understanding how and why software engineering activities should be carried out. They also argue that improving the software process and the resulting prod- ucts with the acquired knowledge is possible. The authors also highlight the main characteristics of a case study, namely 1) their conclusions must reflect a clear chain of evidence, whether qualitative or quantitative, collected from various sources in a planned and consistent manner; and 2) they must add to the existing body of knowledge, based on established theory, if any, or build such theory. Thus, the case study de- scribed below is proposed as a method of evaluating both the case addressed and the TACT instrument (Yin, 2013). 4.1 Research questions The study aims to evaluate TACT preliminarily. To achieve the aim, the research questions (RQ) are defined as follows: • RQ1. How is the organizational climate in the examined agile teams? – RQ1.1. How did working from home affect the or- ganizational climate of the teams for the analyzed dimensions? • RQ2. How do leaders perceive TACT? • RQ3. Which are the most influential items in each di- mension for the analyzed case? During the planning and execution of the study, teams allo- cated in the same physical environment were working from home due to the COVID-19 pandemic described in Davis et al. (2020). To investigate whether this fact could have im- pacted the organizational climate of the studied teams, we defined RQ1.1. 4.2 Description of the organization and teams The organization analyzed in the study is a big Brazilian bank with millions of customers. It has dozens of development teams, composed of employees and outsourced collaborators. Each team defines the software development process and can choose traditional (structured and RUP) or agile (Scrum, Kanban, XP) methods, among others defined by the organi- zation. Each team has the freedom to define the scenario and artifacts to be developed as long as it is officially stated to the process sector. Regarding leadership, some teams use the role of Scrum Master, but in others, this role is played by the hierarchical leader of the team. When present, the role of coach facili- tates the understanding and dissemination of good agile prac- tices by the teams. During this time of working from home, the team’s monitoring by the agile leader occurs through the ceremonies that continue to be performed, the monitoring of task execution, and meetings and interactions using Mi- crosoft Teams and corporate Skype resources. Even with the change in the work routine, it was reported that tasks con- tinue to be delivered within the established deadlines and with the required quality. Three teams, named A, B, and C, were selected by conve- nience to participate in the case study. The teams have em- ployees from the organization as well as outsourced mem- bers. 4.3 Data collection For the data collection, we used interviews, document analy- sis, and the application of TACT. Data collection took place between January 2020 and March 2021. The first data collection procedure was an interview with a process coordinator of the organization. The objective was to understand how the company assessed the organizational cli- mate, which difficulties were faced with assessing the orga- nizational climate of agile software development teams, how the development process was like, and how the composition of agile teams was like. The second procedure was to analyze the executive reports with the last two organizational climate assessments results. It is noteworthy that the assessment performed by the organi- zation is biennial and does not consider the team in which em- ployees are allocated. Only employees and superintendence department. For this reason, it is not possible to understand the climate of individual teams. The third procedure was the assessment of the organi- zational climate in the teams through TACT. All members of teams were invited to participate voluntarily and anony- mously in the study. The organizational climate survey was applied in three cycles called pulses. Table 1 shows the di- mensions applied to each team by pulse and the number of participants by each team in each pulse. Pulse 1 was executed in June 2020 for Team A and B. Pulse 2 was executed in February 2021 for Team C. Lastly, Pulse 3 was executed in March 2021, and all teams participated. The numbers in the columns Team A, Team B, and Team C represent the size of each team at the moment each pulse was executed. In the pe- riod between pulse 1 and pulse 2, some team members from Team A and B were allocated to other teams due to the con- clusion of the product module. Table 1. Measurement cycles Pulse Dimension Date Team A Team B Team C 1 Communication, Collabo- ration, Leadership Jun20 13 10 - 2 Communication, Collabo- ration, Leadership Feb21 - - 4 3 Autonomy, Decision- making, Client Involve- ment Mar21 9 5 4 In addition to the items present in Appendix A.2, three TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 open-ended questions were introduced: “Regarding the ex- amined dimensions (communication, collaboration, leader- ship, autonomy, decision-making, and client involvement), what are the main challenges for your team at this time work- ing from home?” and “Do you have anything to add about your team’s organizational climate?”. In addition, at the be- ginning of the instrument, we included a description with the definition of the organizational climate and the objective of the assessment. Next, we presented a consent form to comply with ethical principles in which we informed that participa- tion would be anonymous, voluntary and that the participant could abandon the assessment at any time without penalties. The fourth procedure represents the execution of semi- structured interviews with the leaders of the respective teams. These interviews were designed to present the results of the climate assessment and capture the leader’s perception of TACT and the team’s organizational climate. To do this, they were asked some questions such as “How do you evaluate the results, by dimension, of the organizational climate as- sessment carried out by the team? Do the results by dimen- sion represent your perception of the team’s daily life? In your opinion, was there any result that surprised you? Do you believe that the items used represent expected behaviors in agility (mindset, values, principles, and practices)? Oth- erwise, explain why the item does not represent expected be- havior”. At the end of the interview, we sent a link to the leader to evaluate TACT through TAM (Technology Acceptance Model) (Venkatesh and Davis, 2000; Venkatesh and Bala, 2008). The dimensions of Intention to Use, Perceived Use- fulness, and Output Quality were used (Venkatesh and Davis, 2000; Venkatesh and Bala, 2008). In the interviews, we used a consent form to present and assure ethical aspects. 5 Case Study Results This section aims to present the results of the organizational climate assessment, thus answering the research questions. 5.1 How is the organizational climate in the ex- amined agile teams? (RQ1) Teams were allowed to answer the survey for 8 days on each pulse. We checked the data and calculations performed by TACT. In total, 22 team members participated in Pulse 1, 12 out of 13 (i.e., 92.31% of members) from Team A and 10 (100%) from Team B. In Pulse 2, 3 out of 4 (75%) members from Team C answered the survey. On the last pulse, 4 out of 9 (44%) members from Team A, 5 (100%) members from Team B, and 4 (100%) members from Team C participated in the study. Table 2 shows the frequency for each investigated dimen- sion. The “Dimension” column represents the description of the dimension. For each team, the relative frequencies (count for each value assigned by the members) and absolute fre- quencies (percentage in parentheses) were calculated accord- ing to the aforementioned Likert scale. In Table 2, we chose to count the values “strongly agree” and “agree” in the col- umn “Positive”, and “strongly disagree” and “disagree” in the column “Negative”. Finally, we consider the frequency of “neutral” to categorize the organizational climate as neu- tral. Figure 2 shows the TACT dashboard, which is used to present the results of the climate assessment. The climate is classified as positive, neutral, or negative to facilitate the analysis of results by team members, leaders, and others in- volved. When analyzing the results in Table 2, higher frequencies can be observed in the “Positive” column for Team B and C in all dimensions. Considering that the 49 items represent good behavior expected by the main existing roles in an agile team, it is possible to classify the organizational climate of Teams B and C as positive or favorable for all dimensions. In team A, the organizational climate can be classified as i) positive for the Communication, Collaboration, and Leader- ship, and ii) negative for Autonomy, Decision-Making, and Client Involvement dimensions. Table 2 shows that Team B and C presented a positive cli- mate superior to that of Team A in all dimensions. For exam- ple, the frequency of the Communication dimension was 82 (91.1%) for Team B, 20 (74%) for Team C, and 62 (58%) for Team A. Neutral and negative results represent points of attention for an analysis of possible causes and impacts on involved roles, elements of the process, the development project, or the team’s culture in general. 5.1.1 Analysis of Organizational Climate from Team A Among the assessed teams, Team A showed more items eval- uated as negative and neutral (see Table 2). Thereby, the Organizational Climate can be considered negative for the Autonomy, Decision-making, and Customer Involvement di- mensions. However, it is observed (i) positive evaluations in the items referring to the interaction between the team mem- bers, and (ii) negative and neutral evaluations in the interac- tions that involve the product owner and the leader. Some points of attention were clarified in open-ended questions and the interview with the leader. In response to the question about the challenges for the Communication dimension at this time of working from home, a member of Team A said that “virtual rooms, when poorly managed, end up providing a space for inopportune conversations”. This statement was also corroborated by the leader of Team B “they think they talk too much, lose focus a little bit”, mentioning the feedback obtained from the team at the previous day’s daily meeting. These comments can be associated with item “IT04. Team mem- bers frequently talk about club, entertainment, gym, parties, sports, and films”. For item “IT07. In the current project, the daily meeting allows to know project problems and team dif- ficulties”, the leader of Team A admitted the negative result, “the team decided not to hold the daily meeting during the period of working from home anymore, the difficulties are addressed by WhatsApp and the virtual room at Microsoft Teams”. In addition, the leader of Team A agreed with the team, noting the negative result for item “IT02. The team keeps the list of impediments, risks, and control actions up- dated” “many times I have to register the impediments myself, TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Table 2. Results of the organizational climate assessment for teams A, B and C Team A Team B Team C Dimension Negative Neutral Positive Negative Neutral Positive Negative Neutral Positive Communication 23 (21%) 23 (21%) 62 (58%) 6 (7%) 2 (2%) 82 (91%) 1 (4%) 6 (22%) 20 (74%) Collaboration 10 (10%) 20 (20%) 66 (70%) 0 (0%) 2 (3%) 78 (97%) 0 (0%) 2 (7%) 22 (93%) Leadership 27 (23%) 29 (24%) 64 (54%) 0 (0%) 13 (13%) 87 (87%) 0 (0%) 5 (17%) 25 (83%) Autonomy 12 (43%) 9 (32%) 7 (25%) 2 (6%) 2 (6%) 31 (88%) 0 (0%) 2 (7%) 26 (93%) Decision- making 9 (28%) 16 (50%) 7 (22%) 0 (0%) 4 (10%) 36 (90%) 0 (0%) 3 (9%) 29 (91%) Client Involve- ment 6 (22%) 11 (39%) 11 (39%) 0 (0%) 2 (6%) 33 (94%) (0%) 0 (0%) 28 (100%) Figure 2. Part of TACT’s Dashboard (Pulse 1: Team A Results) they don’t do it”. In the Pulse 3, the item “IT39. My team has open and effective communication” had all neu- tral assessments (4 100%), reflecting a change in the team’s climate for Communication dimension. Team A showed a greater positive climate in relation to the collaboration between the members themselves, for example, in items “IT10. Team members consider sharing know-how with each other” and “IT12. My team works efficiently to- gether when in the face of difficulties”. However, when col- laboration involves the product owner and the leader, points of attention in the item “IT17. In the current project, the team, the product owner, and the team facilitator work excellently together to plan the iteration” deserve to be stressed. With the analysis of the open question “Do you have anything to add about your team’s organizational climate?”, it was possi- ble to identify potentials causes for the negative assessment for item IT17. The members reported that “after the coor- dination change occurred, there was some distancing between the PO and the team” and “the team leader does not play her role”. This assessment of the negative climate was repeated in Pulse 3. During the interview, in the analysis of IT17, the leader of Team A stated that “the team often wants to impose on the PO what they think should be implemented in the product, they feel like they own the product”. The leader also pointed out that “the employee designated as PO cannot develop stories at team speed. Often, the product owner cannot approve a Sprint with the business customer, as customers have other priorities, which compromises the next Sprint planning”. RegardingtheAutonomyandDecision-makingdimension for iteration planning, the leader reported “sometimes there are demands that override all planning. We lived this re- cently, every time an unplanned demand arrived that passed over all others demands. This hinders the planning team’s autonomy”. The leader also declared “these past few months have been hard, a little stressful. Most of the demands were out for planning”. The comments were said by the leader in the analysis of items “IT34. My team has the decision author- ity and responsibility to plan the iteration” and “IT35. My team has time to plan the changes without excessive stress or pressure”. The climate can be characterized as negative to Decision- making dimension, considering 9 (28%) items assessed as negative and 16 (50%) items as neutral. The item “IT41. The dependencies between the tasks do NOT hinder the fluid- ity of the project and do NOT cause major restrictions” ob- tained 75% of negative evaluations. About Decision-Making, a member of Team A reported: “the decision-making process is still not very participatory”. In the analysis of item IT41, the leader of Team A stated “the dependencies between the tasks are getting in the way. Demands have a number of tasks that impact. If the PO does not approve the changes, this cre- ates a configuration and change problem. In the company, if you put a demand into production and do not validate with the PO, the infrastructure team rollback the demand”. With respect to Client Involvement, a member from Team A described “business representatives, fail to fulfill their role during homologation, impacting the delivery in production not only that specific demand but also many others, as they depend on the implementation of the first demand”. 5.1.2 Analysis of Organizational Climate from Team B In Team B, more than 90% of the items were positively eval- uated. However, some items were evaluated as negative and neutral, thus representing points of attention. About the Communication dimension, a team member re- ported “communication continues to flow very well, keep- ing productivity high and positive”. Another report showed the good climate for Collaboration between team members “when there is some difficulty to identifying an error in the tests, we share the screen, we make audio calls, we include other team members, whom we know have some more spe- TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 cific experience at that point, in the conversations”. Team B leader did not obtain any negative evaluations, only 3 neutral in the item “IT25. The team facilitator gives the team helpful feedback on how to be more agile”. Several praises for the performance of the agile leader during the pe- riod of working from home were registered in response to the open questions. These reports include “... his work remained close and very positive”, “... considering different points of view”, “... moving together, even at a time of working from home”. Regarding the Autonomy dimension, the team leader said “the team autonomy is very good. The members are partic- ipatory. In the team, there is no expression ‘this is my task, or this is not my problem’ ”. A member of Team B wrote, “team members have always been autonomous about their tasks within each user story developed”. Concerning the Decision-making dimension, a member of Team B reported dissatisfaction “the main challenges are when the team’s decisions come up against approval from other areas”. Analyzing the item “IT35. My team have time to plan the changes without excessive stress or pressure”, the leader reported “In the last few months, we had several PO changes in the projects. Before, the PO was of IT , now by determination of the company the PO is of the business. The new PO does not ‘walk’ with the team. She does not feel part of the team. She did not want to be a PO. As the PO was not planning , the team had to plan it”. The previous problems reported by Team B leader may have influenced the two neutral evaluations (2 6%, see Table 2) recorded in the dimension Client Involvement. The items “IT44. In the current project, there are frequent meetings with business representatives and the team” and “IT47. The cur- rent project does NOT have frequent requirement changes due to bad user stories definition” had neutral evaluations. Analyzing items IT44 and IT47, the Team B leader reported “many times the team had to prioritize and refine the user stories without the participation of the PO. After planning, she made several changes to the user stories and the acceptance criteria”. 5.1.3 Analysis of Organizational Climate from Team C Regarding the Communication and Collaboration dimension, Team Leader C said “The team is new. They have only 3 months in this project. They already knew each other. We have an excellent interaction. I do not know the team personally. What gets in the way are limi- tations of the tool (Microsoft Teams) because they do not have full access. But the collaboration between them is excellent”. Regarding all neutral (3 100%) assess- ments in the item “IT05. During the retrospectives, the team finds the best way to do things”, the leader reported “We still have not managed to do the retrospective meetings formally, the team is new. The team started by resolving only incidents. We talked, but not formally at a ceremony”. Regarding the 3 neutral assessments involving iteration planning items “IT34. My team has decision authority and responsibility to plan the iteration” and “IT35. My team has time to plan the changes without excessive stress or pres- sure”, the leader said “They have autonomy. In the current project, they managed to negotiate changes in user stories. They had the autonomy to adjust the planning”. About the pressure in Team C, the leader commented “It should also be considered that the product under development has a fixed date (which cannot be changed) to be launched. The product impacts millions of bank customers”. Analyzing the Decision-making dimension, a member of Team C wrote “decision-making is shared between the mem- bers outsourced, the members of the company, and the busi- ness representatives. We can all contribute with equal weight. Working from home facilitated the engagement and collabo- ration between these 3 roles”. On Autonomy dimension, an- other member wrote “the autonomy limits are agreed with the client”. Despite 100% positive evaluations of the Client Involve- ment dimension, one member reported that the Product Owner was not allocated in the same virtual environment. “In some moments, communication with the management area is not so synchronous, as we do not have access to the same communication tool (Microsoft Teams), but the continuous meetings in this same tool make it easier to exchange information and questions”. The 100% positive assessment of the team in the Client In- volvement dimension did not surprise the leader. The leader declared “The managers praise the team a lot. In these last weeks, the managers have stayed together for up to 4 hours doing the backlog refining. I have never seen such engage- ment. In this project, there are many stakeholders involved. At this time of working from home, they are available to answer questions over the phone. Now, we are currently holding 1-hour refinement meetings. The re- port used at the demonstration meeting containing the evi- dence was highly praised by the PO. The PO said: ‘I never got a homologation script with evidence that did not have errors’ ”. 5.1.4 How did working from home affect the organiza- tional climate of the teams for the analyzed dimen- sions? (RQ1.1) Team members reported some challenges that could have im- pacted the organizational climate as they adapted to the pe- riod of working from home. The challenges mentioned were difficulty with communication tools; infrastructure prob- lems; difficulties in reaching the support team; managing in- opportune conversations in virtual rooms; absence of the fa- cilitator at the ceremonies; customer contract hinders the ac- tion of the facilitator; and other challenges already present before the period of working from home. Regarding the Communication dimension, members of Team A reported that “working from home actually facil- itated team communication” and that there has been “im- proved contact while working from home, we communicate more”. In Team B, the statement “our team is managing to maintain a good dialogue to clarify project issues” stands out. The challenges identified for this dimension mention the network infrastructure and supporting software. In relation to the Collaboration dimension, the challenges TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 captured in the open-ended questions point to Team B’s col- laboration difficulties with the external support team “there have been challenges, some of which required the involve- ment of the support team”. One member reported a prefer- ence for working in person with the team: “... but I believe that being in the same physical space, help, and assistance would sometimes flow better”. Another member stated that “the challenges are the same as they were before working from home”. Regarding the Leadership dimension, no issues were no- ticed in the performance of the leaders of Team B and C. On the other hand, members of Team A reported the absence of the team’s leader in ceremonies and a certain distance from the team’s activities. In general, the members’ responses did not indicate changes in organizational climate due to working from home for the dimensions investigated on TACT. 5.2 How do leaders perceive TACT? (RQ2) During the interviews, for each analyzed dimension, the fol- lowing question was asked to the leaders: “Do you believe that the items used represent expected behaviors in agility (mindset, values, principles, and practices)? Otherwise, ex- plain why the item does not represent expected behavior”. Regarding this question, no item was assessed as not being consistent with agility. In the final stage of the interview, the following questions were asked “In your opinion, what are the benefits of using this instrument?” and “How can the organizational climate assessment tool be improved?”. In relation to the first ques- tion, the leader of Team A answered “I found it interesting, you can map out what needs attention... I can notice other things, interesting... It exposes, gives you a view of what is happening. Very practical, because we can focus on the point that needs attention”. The Leader of Team B agreed, saying “I was able to see the positive things and the neutral points in order to try to improve... The visual formatting (graph- ics) was very clear. I managed to understand the results ef- fortlessly”. The leaders did not report any suggestions to im- prove the instrument. After the interview, TAM (Venkatesh and Davis, 2000; Venkatesh and Bala, 2008) was used, through the dimensions of TAM for the leaders to evaluate TACT. Some items taken into consideration in the assessment, for example, were “As- suming I have access to the instrument, I intend to use it”, “Using the instrument improves my performance in my job”, and “The quality of the output I get from the system is high”. Considering a 7-point Likert scale, most leaders’ responses were the options “Somewhat agree” and “Strongly agree” for all items of dimensions of Intention to Use, Perceived Use- fulness, and Output Quality. 5.3 Which are the most influential items in each dimension for the analyzed case? (RQ3) Due to the large number of items defined on TACT, it is rele- vant to identify the most important items for this case study, i.e., the most influential items in each dimension. For this pur- pose, we performed Factor Analysis (FA). FA is commonly used in Software Engineering to analyze items that use the Likert Scale (Sharma and Gupta, 2012; Klünder et al., 2020; Graziotin et al., 2020). Graziotin et al. (2020) assert that FA allows to reduce the dimensionality of the problem space (i.e., reducing factors and/or associated items) and explaining the variance in the observed variables. In the case of analyses intended to assess a single construct, factor analysis helps identify those items that (best) represent the construct we are interested in, so that we can exclude the other items (Graziotin et al., 2020). The quantitative results were processed using the R tool (v. 4.0.2) using primarily the psych library (Revelle, 2018). It should be stressed that these procedures have an initial ex- ploratory purpose and are not conclusive, as the small sam- ple size (N = 25 - Pulse 1 and 2; N = 13 - Pulse 3), non- randomness and data distribution can have interfered with the results (Dima, 2018; Kyriazos, 2018). The adopted pro- cedures were i) analysis frequency of variation of the items and correlation matrix and ii) Factor Analysis. In step one, the response frequencies for all items are checked to verify whether the items have enough variation to differentiate respondents. If an insufficient variation is iden- tified (i.e., 95% of responses in a single category for an ordi- nal item), the item needs to be excluded from further analysis (Dima, 2018). In this case study, no items needed to be ex- cluded. To continue the analysis of step one, the item correlations (see Figure 3) were plotted for an initial visual diagnosis of the items and structure of the TACT dimensions (Dima, 2018). A higher degree of association between items of the same dimension may already be visible in the correlation matrix (Figure 2). Negative associations between items may indicate the need for reverse item coding, while items with weak associations consistent with other items may prove to be non-scalable in later stages (Dima, 2018). Analyzing the Spearman correlations (ρ) for the test case (Figure 3), we can observe: i) absence of negative correla- tions; ii) IT04 and IT32 with insignificant positive correla- tion, thus IT04 and IT032 will be excluded from next ana- lyzes; and iii) in general, high and moderate positive corre- lation between items in the dimensions. Critical values of ρ (0.9 to 1 - very high; 0.70 to 0.90 - high; 0.51 to 0.70 - mod- erate; 0.31 to 0.5 – low; and 0 to 0.3 - insignificant) (Hinkle et al., 2003). To start the second step, we performed the test of calculat- ing the Kaiser-Meyer-Olkin index (KMO). The KMO index is a statistical test that suggests the proportion of variance of the items that may be explained by a latent variable. The values KMO (see Table 3) were considered appropriate for the FA in each dimension. Value of KMO ( < 0.5 - unaccept- able; > 0.5 and < 0.7 - mediocre; > 0.7 < 0.8; good; 0,8 e 0,9 excellent) (Field et al., 2012). As indicated by Field et al. (2012), the next analysis was conducted on the polychoric correlation matrix. We used the Parallel Analysis graph (Horn, 1965) to investigate the plau- sibility of the initial model proposed on TACT, i. e., the asso- ciation of the items with their dimension. Figure 4 shows the Parallel Analysis graph (x-axis displays the “Factor Number” TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Figure 3. Correlation matrix of the dimensions Figure 4. Parallel Analysis graphic TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Table 3. Quantitative analysis results Dimension Item λ Communication IT03 0.823 IT05 0.760 IT09 0.703 IT01 0.686 IT07 0.682 IT08 0.606 KMO = 0.75 IT06 0.603 α = 0.9 IT02 0.543 Collaboration IT12 0.883 IT15 0.842 IT10 0.800 IT13 0.694 IT11 0.632 IT14 0.622 KMO = 0.67 IT17 0.618 α = 0.9 IT16 0.579 Leadership IT21 0.858 IT19 0.813 IT20 0.745 IT23 0.739 IT27 0.728 IT24 0.721 IT22 0.698 IT26 0.668 KMO = 0.85 IT18 0.665 α = 0.97 IT25 0.585 Autonomy IT29 0.827 IT28 0.702 IT33 0.680 IT30 0.610 KMO = 0.63 IT31 0.553 α = 0.95 IT34 0.514 Decision-making IT39 0.827 IT36 0.672 IT42 0.651 IT37 0.625 IT40 0.563 IT38 0.547 KMO = 0.7 IT41 0.491 α = 0.94 IT35 0.387 Client Involvement IT45 0.792 IT46 0.746 IT43 0.731 IT44 0.731 IT49 0.720 KMO = 0.74 IT48 0.452 α = 0.94 IT47 0.337 and y-axis represents the “Eigenvalue”). As per the Kaiser criterion, only factors with eigenvalues greater than 1 can be retained (Kaiser, 1960). The data simulated by the Parallel Analysis confirmed the hypothesis of retaining one factor by dimension. As shown in Figure 4, all dimensions can be ex- plained by a single factor. The FA was performed separated for each dimension to verify the more significant items. Table 3 shows the quanti- tative results. Analyzing the column “Dimension” (Table 3) and the first line “Communication”, it is possible to verify that the items are ordered by significance. The Factor load- ing (λ) (third column of Table 3) indicates the correlation of the item for the associated dimension. Regarding the small sample size, Field et al. (2012) argue that if a factor has four or more loadings greater than .6 then it is reliable regardless of sample size. Analyzing the Communication dimension (Table 3), the items IT03 (λ = 0.823) and IT05 (λ = 0.760) have the high- est factor load, and they can be considered the most signifi- cant ones. Therefore, for effective communication, the team should consider empathic listening (IT03) and ensure the nec- essary discussions on possible decision-making agreed dur- ing the retrospectives (IT05). For the Collaboration dimen- sion, items IT12 (λ = 0.884) and IT15 (λ = 0.842) have the highest factor load, and they can be considered the most rel- evant. The IT12 represents that the team should work effi- ciently together to solve problems, and the IT12 the collabo- ration to innovation. IntheLeadershipdimension, itemIT21ishighlighted.The item IT21 (λ = 0.858) measures the activities of the team leader in discussing the problems and impediments of the team. The facilitator’s behavior in protecting the team au- tonomy from external interference IT29 (λ = 0.827) has a high correlation to other items to the Autonomy dimension. For effective Decision-making, the teams should have open and effective communication IT39 (λ = 0.827). Lastly, for di- mension Client Involvement, the item IT45 (λ = 0.792) rep- resents the opportunity of stakeholders to suggest changes or improvements to the software. We calculated the reliability (see Table 3) of TACT dimen- sions using the α-Cronbach coefficient (Landis and Koch, 1977). The α-Cronbach indexes for each dimension are α > 0.8, which implies the reliability of TACT for this case study is high (Landis and Koch, 1977). 6 Discussion 6.1 Case Study We execute a case study to assess TACT preliminarily. TACT has 49 items to assess the climatic dimensions of Commu- nication, Collaboration, Leadership, Autonomy, Decision- Making, and Client Involvement in agile development teams. The case study was carried out with three teams working at a bank. The climate assessment took place during a period in which teams that were previously physically allocated to- gether were instead conditioned to work from home. In addi- tion to the items established in TACT, open-ended questions were used to understand the challenges faced by members working from home. In the end, we conducted interviews with the leaders to understand the possible causes or impacts of the items evaluated. Analyzing the frequency of responses attributed to the items by the members, the answers to the open questions and the data from the interviews, there are signs of a positive or- ganizational climate in Teams B and C. On the other hand, there are signs of a negative organizational climate in Team A. Thereby, negative and neutral frequencies were observed TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 in some items, which can represent points of attention. Communication, collaboration, autonomy, and decision- making are critical human factors in agile software devel- opment teams because members use them to plan and exe- cute iterations, besides periodically adjusting the process or the team’s behavior (Chagas et al., 2015; PMI and Agile Al- liance, 2017). Regarding the Communication, Collaboration, Autonomy, and Decision-making dimensions, there were positive frequencies for the relationship between the mem- bers of each team (e.g., IT03, IT11, IT12, IT28, IT31, and IT38). However, negative and neutral frequencies point to possible difficulties in Team A when collaboration, commu- nication, autonomy, and decision-making involve the roles of the product owner (IT33 and IT49) and the facilitator (IT09, IT17, and IT29) and, also, agile ceremonies (IT08 and IT34) and artifacts (IT02 and IT08). Team A abandoned or mischaracterized some agile prac- tices while working from home, for instance, the daily meet- ing (IT07). Regarding the artifacts, the team was not com- municating some impediments to the leader (IT02) and both the product owner and the team were not adapting the re- quirements for the user story format (IT08 and IT16), due to the contract with the software factory, which established the requirement in another format for payment estimates. An- other critical factor identified in Team A was the inability of the product owner to establish requirements according to the team’s speed and capacity (IT17). Thus, although the col- laboration between team members was classified as positive in Team A, the relationship with the product owner and the team facilitator reflected points of attention, which can be ob- served in the statement from one member: “the agile method- ology is being abolished in our team”. As previously stated, leadership is one of the central el- ements for forming the organizational climate (Schneider et al., 2014). The main activities of the servant leader can be summarized in (i) remove team impediments and (ii) facilitate, disseminate, and ensure the use of agile values, practices, and rules (Noll et al., 2017; PMI and Agile Al- liance, 2017). Concerning leadership, during the interview, the leader of Team A clarified the negative assessment for item IT19, “I follow it closely when I am called, when I am needed”. In Team B, TACT captured a closer relationship be- tween the leader and the team. However, when the leader of Team B analyzed the neutral points of IT19, she made the fol- lowing statement: “I have not been able to dedicate myself, to be the Scrum Master that I was [before working from home]. The agility factor has been the greatest challenge, solving impediments faster. I need to do things that I still have not managed to”. The challenges captured in several reports did not point out new insights about working from home for the dimen- sionsinvestigated.Concerningthechallenges, theteammem- bers reported “The challenges are the same as those that ex- isted before working from home”, “There are no new prob- lems in working from home. They [the challenges] existed before”, and “the current moment of working from home has not brought any new challenges so far(...)”. It is worth not- ing that, according to the report by the process coordinator, the quality and performance indicators are the same as before working from home began. Supporting the report of the pro- cess coordinator, Serrador et al. (2018) claim that it is often argued that teams allocated in the same physical space have a better performance, a greater success in the project. However, the authors also did not identify a significant difference be- tween local and remote teams in the study on the climate for the success of development projects (Serrador et al., 2018). 6.2 Preliminary evaluation of TACT The literature recommends implementing a pilot study for the initial assessment of instruments that measure behaviors, attitudes, or feelings (Dybå, 2000; Patterson et al., 2005; Shahzad et al., 2017; Recker, 2013). The pilot must utilize a sample with the same characteristics as the target popula- tion (Anderson and West, 1998; Dybå, 2000; Shahzad et al., 2017; Patterson et al., 2005; Recker, 2013). On TACT, we decided to carry out the preliminary assessment through a casestudybecausewewantedtocapturetheperceptionofthe teams’ climate in different data sources. The results and anal- ysis presented in the previous sections established a chain of evidence that allows us to infer that TACT can capture the context of organizational climate experienced in the teams. In the evaluation by specialists (see Section 3.3), every TACT item related to agile software development teams was considered. This assessment is already evidence of the con- tent validity of TACT. In a qualitative analysis (see Section 5.2), the leaders con- firmed that the items represent agile values, principles, and practices. Through the dimensions of TAM (Venkatesh and Davis, 2000; Venkatesh and Bala, 2008), leaders rated TACT positively for Intention to Use, Perceived Usefulness, and Output Quality. In quantitative analysis, the correlation matrix (Figure 3) revealed a high and moderate positive correlation between most of the items of each dimension. Only the items IT04 and IT32 showed an insignificant correlation with the other dimension items. Thus, we excluded IT04 and IT032 from the Factor Analysis. Development teams that talk about the subject of IT04 reported a positive emotion, contributing to the group’s optimism (Licorish and MacDonell, 2014). However, Team A members understood that talking about these issues would be a negative behavior when Team A analyzed the results. This misinterpretation may have been caused by the description of the item “IT04. Team members FREQUENTLY talk about club (...)”. Regarding IT04, leader Team C said, “perhaps the word ’frequently’ caused the mis- understanding”. Considering quantitative analysis and the reports, we excluded IT04 of TACT. We have not captured reports of misinterpretation on item IT32. The low correla- tion may be relative to the sample and not to the construct. Thus, we opted to keep IT32 in TACT. Factor Analysis allowed, based on the response patterns, to verify the proposed structure of TACT, i.e., the associated items in each dimension. The Parallel Analysis graph (Fig- ure 4) indicated that a single factor could explain all dimen- sions. Furthermore, most TACT items have high (> 6) factor loadings (see Table 3). Therefore, there is initial empirical evidence that the structure proposed in TACT is acceptable. Quantitative analysis revealed high reliability of the TACT dimensions (see Table 3). The α-Cronbach indexes for each TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 dimension are α > 0.8, which implies the reliability of TACT for this case study is high (Landis and Koch, 1977). 6.3 TACT use recommendations Wagner et al. (2020) recommend that Software engineering research should either adopt or develop psychometrically val- idated questionnaires. We extend that recommendation to the companies that realize organizational climate assessment. Validating a climate instrument without selling intent is chal- lenging because it is necessary to find companies or persons willing to invest their time answering a questionnaire with- out a counterpart. We highlight that all evidence of validity and reliability are conditioned to the date this research was conducted, i.e., the more investigations are executed using TACT, the more evidence of validity and reliability there will be. Thereby, TACT items can be used by researchers who want to measure proposed constructs or investigate other pos- sible factor structures. The organizational climate is measured through mani- fested behaviors or perceived feelings by the employees. Cli- mate instruments are self-reports. Only the team member knows how he is feeling. Although many factors can skew team member’s views, when several individuals point in the same direction, a point of investigation is revealed. For exam- ple, if an item with too many negative ratings might indicate a lack of practice, a specific problem, or a misunderstanding about the agile mindset. Therefore, climate instruments only allow for a pre-diagnosis of what must be investigated and dealt with in the later stages of the Organizational Climate Management process. Organizational climate instruments measure some latent variables (those that are not directly observable). TACT items represent examples of good behaviors or practices widely used in agile software development teams. There- fore, team leaders, managers, or the responsible for preparing and conducting organizational climate assessments can use the TACT items for a more accurate diagnosis. If a specific item has many neutral or negative evaluations, an investiga- tion point is revealed. For example, the assessment of item “IT35. My team has time to plan the changes without exces- sive stress or pressure” shows how the team member feels (stressed/pressured) and suggests what project activities or situations (such as interaction planning, task estimation, abu- sive or unrealistic deadlines given by PO or manager) might be the cause of that feeling. Notice that terms in the item de- scription (for example, plan, stress, and pressure in IT35) al- lows team members to reflect on how they are feeling about the day-to-day events. To create every TACT items description, we used generic nomenclatures for roles and practices used in hybrid and agile processes. Scrum is the most used agile methodology (Digital.ai, 2020). However, we do not use the names of the roles or ceremonies from Scrum, e.g., we use Team Facilita- tor, Iteraction, and Meeting Review instead of Scrum Master, Sprint, and Sprint Retrospective. By doing that, we expect to reach more teams using different process configurations. Thereby, if TACT is used by a team where the software de- velopment process has other names to roles or ceremonies or still does not have a specific role, the team members can mis- understand the items. To address this limitation and threat, at the beginning of the climate survey, we show the vocabulary of terms used in TACT compared with Scrum terms. Regarding the number of items and time interval of the application of TACT, based on a previous study (Dutra and Santos, 2020), we claim that using many items and having a long time interval in the organizational climate survey in agile teams can hinder the assessment, diagnosis, and estab- lishment of actions to climate management. Having too many items in climate surveys and the lack of control activities can also demotivate the team member’s participation in new cli- mate surveys. In that regard, we recommend that practition- ers adopt one or two dimensions by cycle, performing several cycles per year. However, more critical than measuring the organizational climate is involving the team in discussions of possible actions that allow a climate change. A simple open-ended question that can help team engagement in cli- mate management is “How to improve your team’s organiza- tional climate?”. 7 Limitations and Threats to Validity The research procedures used in this study are adequate to build an organizational climate instrument, but we faced some limitations. The main one concerns the small sample size. As mentioned in Section 5.3, the quantitative proce- dures have an initial exploratory purpose. Due to the small sample size, the use of Factor Analysis (FA) is not possible without segregating the data. Due to that, we conducted FA by each TACT dimension. In future studies (see Section 8.1), we will perform Exploratory and Confirmatory FA. In Pulse 3, only 4 out of 9 Team A members answered the survey. The number of participants can hinder Team A’s organiza- tional climate assessment because the four respondents may have the same perspective of the team organizational climate while the other members of Team A that did not participate in Pulse 3 would have another perspective. The Team leader A interview helped us confirm the results and deal with this limitation. Recker (2013) proposes some principles for evaluating qualitative and quantitative studies. Concerning reliability, a contextual description of the or- ganization was presented as well as direct quotes from team members and leaders which were considered to support the analysis. Thus, it is possible to guarantee that individuals other than the researchers, when considering the same obser- vations or data, will reach the same or similar conclusions (Recker, 2013). From a quantitative point of view, an inves- tigation was carried out to assess the study’s reliability, using descriptive statistics, correlations, and Cronbach’s α coeffi- cient. Thus, the reliability of TACT dimensions for the case study sample is high. To address possible threats to internal validity, we decided to use multiple sources of evidence. The team members as- sessed the organizational climate through the TACT items and open-ended questions. In addition, the leaders’ percep- tions were captured through interviews. In this way, a chain of evidence was established, and the review of the evalua- tion results was assured (which also relates to measurement TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 validity). Regarding TACT, two auditors experienced in ag- ile methods and the leaders in the study assessed whether the item descriptions represented elements of agile values, prin- ciples, and practices. External validity concerns how much and when the re- sults of a study can be generalized to other cases or domains (Recker, 2013). To mitigate this threat, we provide detailed descriptions of the study context. However, Schneider et al. (2014) claim that everything that happens in the organization changes its climate. Thus, it is not possible to guarantee sim- ilar results in another cycle in the same examined teams or even in other teams of the same organization. 8 Final Considerations We presented the initial version of TACT (insTrument to As- sess the organizational ClimaTe of agile teams), designed to measure the dimensions of Communication, Collabora- tion, Leadership, Autonomy, Decision Making, and Client In- volvement. We also presented a case study to evaluate TACT and measure the organizational climate of three agile teams from the same organization. Data collection included TACT results, interviews with team leaders, and answers to open- ended questions by the participants. The sample data revealed a positive organizational climate for all dimensions in teams B and C and negative for Au- tonomy, Decision Making, and Client Involvement dimen- sions for team A. Thereby, some items assessed as negative or neutral indicated points of attention. Through open-ended questions and interviews with leaders, the evaluation carried out through TACT was confirmed and the points of atten- tion were better explored. We identified the abandonment of some agile ceremonies, difficulties in planning the iteration, the inability of the product owner to keep up with the speed and capacity of the team, and even the absence of leadership. Based on the statistical analysis of the data from assessing the organizational climate, there is an initial evidence that the validity and reliability of TACT dimensions are high. 8.1 Future works Besides the TACT dimensions proposed in the present study, we are investigating new constructs: Motivation, Trust, Learning, and Knowledge. Other case studies are being exe- cuted to assess the climate of the same three teams mentioned in this study and other four teams of another organization. After finishing the case studies cycle, we will execute a survey to investigate and validate the factorial structure of all TACT dimensions. We will use Exploratory and Confir- matory Factor Analysis to investigate and confirm the mea- sured dimensions. As a result, TACT dimensions and items will likely be reduced. After conducting the survey, we will have the means to create guidelines for using TACT and in- terpret the results. We also intend to investigate the influence of gender, team size, and team members’ experience on agile methodologies in the organizational climate. Moreover, in the future, there might be some value in digging deeper into an investigation on whether the organizational climate of employees and out- sourced team members differs. Acknowledgements We thank UNIRIO (PPQ-UNIRIO 01/2019 and 04/2020; PPINST- UNIRIO 05/2020) for their financial support. References Açikgöz, A. (2017). The Mediating Role of Team Collabora- tion between Procedural Justice Climate and New Product Development Performance. International Journal of Inno- vation Management, 21(04):1750039. Acuña, S. T., Gómez, M., and Juristo, N. (2008). Towards understanding the relationship between team climate and software quality—a quasi-experimental study. Empirical Software Engineering, 13(4):401–434. Açıkgöz, A. and Gunsel, A. (2016). Individual Creativity and Team Climate in Software Development Projects: The Mediating Role of Team Decision Processes. Creativity and Innovation Management, 25(4):445–463. Açıkgöz, A., Günsel, A., Bayyurt, N., and Kuzey, C. (2014). Team Climate, Team Cognition, Team Intuition, and Soft- ware Quality: The Moderating Role of Project Complexity. Group Decision and Negotiation, 23(5):1145–1176. Açıkgöz, A. and Ö. İlhan, Ö. (2015). Climate and Problem Solving in Software Development Teams. Procedia - So- cial and Behavioral Sciences, 207(20 October 2015):502– 511. Ahmed, S., Ahmed, S., Naseem, A., and Razzaq, A. (2017). Motivators and Demotivators of Agile Software Develop- ment: Elicitation and Analysis. International Journal of Advanced Computer Science and Applications, 8(12):1– 11. Ancona, D. G. and Caldwell, D. F. (1992). Bridging the Boundary: External Activity and Performance in Or- ganizational Teams. Administrative Science Quarterly, 37(4):634. Anderson, N. R. and West, M. A. (1998). Measuring climate for work group innovation: development and validation of the team climate inventory. Journal of Organizational Be- havior, 19(3):235–258. Annosi, M. C., Martini, A., Brunetta, F., and Marchegiani, L. (2020). Learning in an agile setting: A multilevel research study on the evolution of organizational routines. Journal of Business Research, 110:554–566. Askarinejadamiri, Z. (2016). Personality requirements in re- quirement engineering of web development: A systematic literature review. In 2016 Second International Confer- ence on Web Research (ICWR), pages 183–188, Tehran, Iran. IEEE. Bandura, A. (2006). Summary for Policymakers. In Inter- governmental Panel on Climate Change, editor, Climate Change 2013 - The Physical Science Basis, pages 1–30. Cambridge University Press, Cambridge. Beck, K., Beedle, M., and et al. Bennekum, A Van (2001). Manifesto for Agile Software Development. TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Chagas, A. (2015). (In Portuguese) O impacto dos fatores humanos nos métodos ágeis. Chagas, A., Santos, M., Santana, C., and Vasconcelos, A. (2015). The Impact of Human Factors on Agile Projects. In 2015 Agile Conference, pages 87–91, National Harbor, MD, USA. IEEE. Curtis, B., Hefley, W. E., and Miller, S. A. (2009). People Capability Maturity Model (P-CMM ) Version 2.0, Second Edition. Technical report, Carnegie Mellon University. Davis, K. G., Kotowski, S. E., Daniel, D., Gerding, T., Nay- lor, J., and Syck, M. (2020). The Home Office: Ergonomic Lessons From the “New Normal”. Ergonomics in Design: The Quarterly of Human Factors Applications, 28(4):4– 10. Delgado-Rico, E., Carretero-Dios, H., and Ruch, W. (2012). Content validity evidences in test development: An ap- plied perspective. International Journal of Clinical and Health Psychology. Digital.ai, T. (2020). 14th Annual State of Agile Report. Technical report, Digital.ai. Dima, A. L. (2018). Scale validation in applied health re- search: tutorial for a 6-step R-based psychometrics pro- tocol. Health Psychology and Behavioral Medicine, 6(1):136–161. Dutra, E., Diirr, B., and Santos, G. (2021). Human factors and their influence on software development teams - a ter- tiary study. In Brazilian Symposium on Software Engineer- ing, SBES ’21, page 442–451, New York, NY, USA. As- sociation for Computing Machinery. Dutra, E., Lima, P., and Santos, G. (2020). An Instrument to Assess the Organizational Climate of Agile Teams - A Pre- liminary Study. In 19th Brazilian Symposium on Software Quality, pages 1–10, São Luis, Brazil. ACM. Dutra, E. and Santos, G. (2020). Organisational climate assessments of Agile teams – a qualitative multiple case study. IET Software, 14(7):861–870. Dutra, J. S., Fischer, A. L., Nakata, L. E., Pereira, J. C. R., and Veloso, E. F. R. (2012). The use categories as indicator of organizational climate in Brazilian companies. Revista de Carreiras e Pessoas, 2:145–176. Dybå, T. (2000). Instrument for measuring the key factors of success in software process improvement. Empirical Software Engineering, 5:357–390. Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10):833–859. Field, A., Miles, J., and Field, Z. (2012). Discovering Statis- tics Using R. SAGE Publications, London, 1 edition. Franca, A., Gouveia, T., Santos, P., Santana, C., and da Silva, F. (2011). Motivation in software engineering: a system- atic review update. In 15th Annual Conference on Eval- uation and Assessment in Software Engineering (EASE 2011), pages 154–163, Durham, UK. IET. Ganesh, M. P. (2013). Climate in software development teams: Role of task interdependence and procedural jus- tice. Asian Academy of Management Journal. Ganesh, M. P. and Gupta, M. (2006). Study of Virtualness, Task Interdependence, Extra-Role Performance and Team Climate in Indian Software Development Teams. Proceed- ings of the 20th Australian New Zealand Academy of Man- agement (ANZAM) Conference on Management, Pragma- tism, Philosophy, Priorities, Central Queensland Univer- sity, Rockhampton, 20:1–19. González-Romá, V., Fortes-Ferreira, L., and Peiró, J. M. (2009). Team climate, climate strength and team perfor- mance. A longitudinal study. Journal of Occupational and Organizational Psychology, 82(3):511–536. Graziotin, D., Lenberg, P., Feldt, R., and Wagner, S. (2020). Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines. ACM Trans. Softw. Eng. Methodol., I(1):Article 111 – 49 pages. Grobelna, K. and Stefan, T. (2019). The Impact of Organiza- tional Climate on the Regularity of Work Speed of Agile Software Development Teams. Entrepreneurhip and Man- agement, 12(1):229–241. Hinkle, D., Wiersma, W., and Jurs, S. (2003). Applied Statistics for the Behavioural Sciences. Houghton Mifflin, Boston, 5 edition. Hoda, R., Kruchten, P., Noble, J., and Marshall, S. (2010). Agility in context. ACM SIGPLAN Notices, 45(10):74– 88. Hohl, P., Klünder, J., van Bennekum, A., Lockard, R., Gif- ford, J., Münch, J., Stupperich, M., and Schneider, K. (2018). Back to the future: origins and directions of the “Agile Manifesto” – views of the originators. Jour- nal of Software Engineering Research and Development, 6(1):15. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2):179–185. Jia, J., Zhang, P., and Capretz, L. F. (2016). Environmen- tal factors influencing individual decision-making behav- ior in software projects. In Proceedings of the 9th Inter- national Workshop on Cooperative and Human Aspects of Software Engineering, pages 86–92, New York, NY, USA. ACM. Kaiser, H. F. (1960). The Application of Electronic Com- puters to Factor Analysis. Educational and Psychological Measurement, 20(1):141–151. Karhatsu, H., Ikonen, M., Kettunen, P., Fagerholm, F., and Abrahamsson, P. (2010). Building blocks for self- organizing software development teams: A framework model and empirical pilot study. In ICSTE 2010 - 2010 2nd International Conference on Software Technology and Engineering, Proceedings. Kettunen, P. (2014). Directing High-Performing Software Teams: Proposal of a Capability-Based Assessment Instru- ment Approach. In Bergsmann J., editor, Lecture Notes in Business Information Processing, chapter Model-Base, pages 229–243. Springer, Cham. Klünder, J., Karajic, D., Tell, P., Karras, O., Münkel, C., Münch, J., MacDonell, S. G., Hebig, R., and Kuhrmann, M. (2020). Determining Context Factors for Hybrid De- velopment Methods with Trained Models. In Proceedings of the International Conference on Software and System Processes, pages 61–70, New York, NY, USA. ACM. Kyriazos, T. A. (2018). Applied Psychometrics: Sample Size and Sample Power Considerations in Factor Analy- sis (EFA, CFA) and SEM in General. Psychology. TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Landis, J. R. and Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159. Lee, J.-N. (2001). The impact of knowledge sharing, organi- zational capability and partnership quality on IS outsourc- ing success. Information and Management, 38(5):323– 335. Lenberg, P., Feldt, R., and Wallgren, L. G. (2015). Be- havioral software engineering: A definition and system- atic literature review. Journal of Systems and Software, 107(September 2015):15–37. Licorish, S. A. and MacDonell, S. G. (2014). Understand- ing the attitudes, knowledge sharing behaviors and task performance of core developers: A longitudinal study. In- formation and Software Technology, 56(12):1578–1596. McAvoy, J. and Butler, T. (2007). The impact of the Abilene Paradox on double-loop learning in an agile team. Infor- mation and Software Technology, 49(6):552–563. Miller, G. J. (2020). Framework for Project Management in Agile Projects: A Quantitative Study. Misra, S. C., Kumar, V., and Kumar, U. (2009). Identifying some important success factors in adopting agile software development practices. Journal of Systems and Software. Moe, N. B., Dings, T., and Dyb, T. (2008). Understanding Self-Organizing Teams in Agile Software Development. In 19th Australian Conference on Software Engineering (aswec 2008), pages 76–85. IEEE. Moe, N. B. and Dingsøyr, T. (2008). Scrum and team effec- tiveness: Theory and practice. In Lecture Notes in Busi- ness Information Processing. Moe, N. B., Dingsøyr, T., and Øyvind, K. (2009). Un- derstanding Shared Leadership in Agile Development: A Case Study. In 2009 42nd Hawaii International Confer- ence on System Sciences, pages 1–10. IEEE. Nianfang Ji and Jie Wang (2012). A software project man- agement simulation model based on team climate factors analysis. In 2012 International Conference on Informa- tion Management, Innovation Management and Industrial Engineering, pages 304–308, Sanya, China. IEEE. Noll, J., Razzak, M. A., Bass, J. M., and Beecham, S. (2017). A Study of the Scrum Master’s Role. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformat- ics), pages 307–323. Springer, Innsbruck, Austria. Patterson, M. G., West, M. A., Shackleton, V. J., Dawson, J. F., Lawthom, R., Maitlis, S., Robinson, D. L., and Wal- lace, A. M. (2005). Validating the organizational cli- mate measure: links to managerial practices, productiv- ity and innovation. Journal of Organizational Behavior, 26(4):379–408. PMI,P.M. I.and AgileAlliance,A. A.(2017). Agile Practice Guide. PMI, Pennsylvania, 1st edition. Recker, J. (2013). Scientific Research in Information Sys- tems. Springer Berlin Heidelberg, Berlin, Heidelberg. Revelle, W. (2018). How to: Use the psych package for factor analysis and data reduction. Technical report, Northwest- ern University. Runeson, P. and Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2):131–164. Schneider, B., Barbera, K. M., Schneider, B., and Barbera, K. M. (2014). Summary and Conclusion. In Barbera, B. S. and M., K., editors, The Oxford Handbook of Organiza- tional Climate and Culture, chapter Summary an, pages 1–14. Oxford University Press, New York, NY, USA. Senapathi, M. and Srinivasan, A. (2013). Sustained agile us- age. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering - EASE ’13, page 119, New York, New York, USA. ACM Press. Serrador, P., Gemino, A., and Horner, B. (2018). Creating a climate for project success. Journal of Modern Project Management, May/August:38–47. Shahzad, F., Xiu, G., and Shahbaz, M. (2017). Organiza- tional culture and innovation performance in Pakistan’s software industry. Technology in Society, 51:66–73. Sharma, A. and Gupta, A. (2012). Impact of organisational climate and demographics on project specific risks in con- text to Indian software industry. International Journal of Project Management, 30(2):176–187. Shull, F., Singer, J., and Sjøberg, D. I. (2008). Guide to Ad- vanced Empirical Software Engineering. Springer Lon- don, London. Soomro, A. B., Salleh, N., Mendes, E., Grundy, J., Burch, G., and Nordin, A. (2016). The effect of software engi- neers’ personality traits on team climate and performance: A Systematic Literature Review. Information and Soft- ware Technology, 73(May 2016):52–65. Spector, P. (1992). Summated Rating Scale Construction: An Introduction. Sage Publications, Inc. Stewart, K. J. and Gosain, S. (2006). The moderating role of development stage in free/open source software project performance. Software Process: Improvement and Prac- tice, 11(2):177–191. Stone, R. W. and Bailey, J. J. (2007). Team Conflict Self- Efficacy and Outcome Expectancy of Business Students. Journal of Education for Business, 82(5):258–266. VandenBos, G. R. E. (2017). APA Dictionary of Psychology. Venkatesh, V. and Bala, H. (2008). Technology Acceptance Model 3 and a Research Agenda on Interventions. Deci- sion Sciences, 39(2):273–315. Venkatesh, V. and Davis, F. D. (2000). A Theoretical Exten- sion of the Technology Acceptance Model: Four Longitu- dinal Field Studies. Management Science, 46(2):186–204. Vishnubhotla, S. D., Mendes, E., and Lundberg, L. (2018). An Insight into the Capabilities of Professionals and Teams in Agile Software Development. In Proceedings of the 2018 7th International Conference on Software and Computer Applications - ICSCA 2018, pages 10–19, New York, New York, USA. ACM Press. Vishnubhotla, S. D., Mendes, E., and Lundberg, L. (2020). Investigating the relationship between personalities and agile team climate of software professionals in a tele- com company. Information and Software Technology, 126:106335. Wagner, S., Mendez, D., Felderer, M., Graziotin, D., and Kalinowski, M. (2020). Contemporary Empirical Meth- ods in Software Engineering. Springer International Pub- TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 lishing, Cham. Yin, R. K. (2013). Case Study Research: Design and Meth- ods. Sage Publications, Los Angeles, 5 edition. Zaineb, G., Shaikh, B., and Ahsan, A. (2012). Recommended cultural and business practices for project based software organization of Pakistan for supporting restructuring of functional organization for implementing agile based de- velopment framework in software projects. In 2012 In- ternational Conference on Information Management, In- novation Management and Industrial Engineering, pages 16–20. IEEE. A Appendix A.1 Constructs Construct Communication • Conceptual definition – Frequent communication between project stake- holders is core to agile software development (Chagas et al., 2015; Chagas, 2015). – “The perception of participatory safety could en- courage team members to be open in communicat- ing their ideas with the team, which could other- wise be risky” (Ganesh, 2013). – Vishnubhotla et al. (2018) reported “the ‘in- sider’voices of Scrum practitioners about the soft skills they consider most valued to have by product owner and scrum master. Communica- tion skills and teamwork were most valued for both roles. Besides them, customer orientation was expressed as important for program man- agers, whereas commitment, responsibility, inter- personal and planning skills were considered valu- able for scrum masters”. – “Gap in communication between developer and customer can guarantee the success of the project while in contrast lack of communication skill causes project problems” (Askarinejadamiri, 2016). • Operational definition – Communication is a capability for the team mem- ber (Vishnubhotla et al., 2018). – Communication is an attribute for team (Vishnub- hotla et al., 2018). – The team has formal and informal communication (Dybå and Dingsøyr, 2008). – The team discusses the project and impediments (Moe and Dingsøyr, 2008; PMI and Agile Al- liance, 2017). – The team discusses how to improve the process and the project (Moe and Dingsøyr, 2008; PMI and Agile Alliance, 2017). Construct Collaboration • Conceptual definition – “Team collaboration is a set of functions and activities carried out before, during, and after teamwork to achieve team objectives” (Açikgöz, 2017). – “Customer collaboration over contract negotia- tion” (Beck et al., 2001). – “Communication and collaboration (C&C) are at the heart of agile software development. As the Agile Manifesto states, ‘individuals and interac- tions over processes and tools’and ‘customer col- laboration over contract negotiation. One aspect in C&C is customer cooperation” (Karhatsu et al., 2010). • Operational definition – Team collaboration involves communication and coordination (Karhatsu et al., 2010). – Collaboration involves work as a team with i) the client (or their representative), ii) the team, and iii) others stakeholders (Açıkgöz et al., 2014; Chagas et al., 2015; Vishnubhotla et al., 2018). Construct Leadership • Conceptual definition – The leadership (in agile projects) is based on the role of the servant leader (PMI and Agile Alliance, 2017). – “Team leadership plays a significant role in im- proving interpersonal and group processes within the team. Team leaders who play the role of ‘com- munication integrators’ are very crucial for the success of the team. The team leader should also ensure periodically whether the members are clear with the team objectives and understand their level of agreement with those objectives” (Ganesh and Gupta, 2006). – “Agile software engineering adopts a leadership style that empowers the people involved in the de- velopment process” (Chagas, 2015). • Operational definition – Leadership is played by a formal role (PMI and Agile Alliance, 2017; Noll et al., 2017). – The leader facilitates ceremonies, removes imped- iments, and shields the team from outside interfer- ence (PMI and Agile Alliance, 2017; Noll et al., 2017). – The leader is a “communication integrator” (Ganesh and Gupta, 2006). Construct Autonomy • Conceptual definition – “The autonomy of a team is defined as the ability to continue to operate in its own way without ex- ternal interference. The role of formal authority is redesigned, so that governance and coordination appear to be the outcome of actions of networks, operating without any formal sanction” (Annosi et al., 2020). TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 – “Autonomy refers to the authority and responsi- bility that a team has in their work. It is a signif- icant factor for team effectiveness. A team must have a real possibility to influence relevant mat- ters; otherwise self-organization is more symbolic than real. On the other hand, a team should not be left completely alone. Instead, while manage- ment should give a team substantial freedom, it should maintain subtle control and have regular checkpoints. Three levels of autonomy are exter- nal, internal, and individual. The external refers to the degree that the people outside of a team in- fluence the team’s decisions. Moreover, it sets the decision-making boundaries for the team. Mean- while, internal autonomy defines how the work is organized inside the team. The team may have sub- stantial power to make decisions while some indi- viduals have none. Great care should be taken to make sure that there really is internal autonomy instead of, for example, team leader autonomy. Fi- nally, individual autonomy, on its part, tells how much an individual has freedom to decide about his or her own work processes” (Karhatsu et al., 2010). • Operational definition – Individual, internal, and external autonomy (Karhatsu et al., 2010). – The team planning the tasks (Karhatsu et al., 2010). – The leader protects the team (Noll et al., 2017; PMI and Agile Alliance, 2017). – The team has good communication with the client (Moe et al., 2008). Construct Decision-Making • Conceptual definition – “Responding to change over following a plan” (Beck et al., 2001). – “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly” (Beck et al., 2001). – “Software development involves interdependent individuals working together to achieve favorable outcomes, so the decision-making behavior of each individual will influence behaviors of other teammates and the project outcome. Individuals have many chances to make a decision in a de- velopment process. For example, individuals may choose a resolution to deal with a conflict. In ag- ile development, each one makes a decision about effort estimation and gives user story points. Indi- viduals may often independently make ‘work’or ‘shirk’choices in teamwork. Under these condi- tions, different individual decision-making behav- iors will generate different results, which are per- tinent to the success or failure of the project” (Jia et al., 2016). – “Product development teams quite often experi- ence problems, barriers and setbacks during the new product development project, which require an immediate and effective decision process to generate sufficient courses of action. Decision pro- cesses refer to team members’ collective efforts to process knowledge about key task-related com- ponents, emerging issues and problems. Individ- ual creativity represents a possible contribution to the teams to deal with these difficulties. More- over, creativity-based decision processes likely al- low the teams to become more proactive when dealing with emerging issues. Indeed, product de- velopment teams have to think outside the box when making decisions, as well as offer practical solutions for problems that can be implemented beyond organizational constraints. Such a process is characterized by the ability to understand com- plexity, to break through prevailing cognitive pat- terns, and to try new paths when old sets do not work” (Açıkgöz and Gunsel, 2016). • Operational definition – Task identity and significance (Jia et al., 2016). – The member perceives recognition of manage- ment and leadership (Jia et al., 2016). – The team has fast and effective communication (Chagas, 2015; Chagas et al., 2015). – The team plains the project without stress or pres- sion (Jia et al., 2016). – The team shares decision-making (Chagas, 2015; Chagas et al., 2015). – The team autonomy influences decision-making (Chagas, 2015; Chagas et al., 2015). Construct Client Involvement • Conceptual definition – Having a client focus is one of the main aims of an agile team (Karhatsu et al., 2010). – “Customer collaboration over contract negotia- tion” (Beck et al., 2001). – “Agile processes promote sustainable develop- ment. The sponsors, developers, and users should be able to maintain a constant pace indefinitely” (Beck et al., 2001). – “Lack of client involvement is ‘the biggest prob- lem’because Agile [requires] fairly strong client involvement” (Karhatsu et al., 2010). – “Welcome changing requirements, even late in de- velopment. Agile processes harness change for the customer’s competitive advantage” (Beck et al., 2001). • Operational definition – Client satisfaction, collaboration, and commit- ment are features of client involvement. (Jia et al., 2016). – A good relationship with users/clients is a motivat- ing aspect for the team (Franca et al., 2011). – The client (or their representative) provides and elucidates requirements (Dybå and Dingsøyr, 2008). TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 – The client (or their representative) validates the software (Dybå and Dingsøyr, 2008). A.2 The items of TACT by dimension Table 4. Items used to measure the Communication dimension Items Source IT01. In this team, we can freely talk to each other about difficulties we are having Stewart and Go- sain (2006) IT02. The team keeps the list of im- pediments, risks and control actions updated # Anderson and West (1998); Miller (2020); PMI and Agile Alliance (2017) IT03. My opinion is always listened to by my team Anderson and West (1998) IT04. Team members frequently talk about club, entertainment, gym, par- ties, sports, and films # * Anderson and West (1998); Licorish and Mac- Donell (2014); Shahzad et al. (2017) IT05. During the retrospectives, the team finds the best way to do things # Chagas et al. (2015); Chagas (2015); González- Romá et al. (2009) IT06. The team knows the skills and technical expertise of team mem- bers, and they use the skills and tech- nical expertise appropriately and ad- equately # Nianfang Ji and Jie Wang (2012) IT07. In the current project, the daily meeting allows to know project problems and team difficulties # Chagas et al. (2015); Dybå and Dingsøyr (2008) IT08. The team and the product owner always reach consensus on the priority of the user stories by ne- gotiating which bug to fix or func- tionality to add # Chagas (2015); Nianfang Ji and Jie Wang (2012) IT09. In the current project, the team and the product owner always solve the disagreements about the iteration scope # Miller (2020); Noll et al. (2017) # Represents original items Table 5. Items used to measure the Collaboration dimension Items Source IT10. Team members consider shar- ing know-how with each other Lee (2001) IT11. Team members always help each other when there is a need Shahzad et al. (2017) IT12. My team works efficiently to- gether when in the face of difficul- ties Açikgöz (2017); Shahzad et al. (2017) IT13. Team members work together as a whole Anderson and West (1998) IT14. All project-related decisions are applied consistently across to af- fected team members Anderson and West (1998) IT15. The team collaborates to look for new ways to analyze the prob- lems # Patterson et al. (2005); Vishnub- hotla et al. (2018) IT16. The team has excellent ability to design the software based on user stories # Açıkgöz et al. (2014); PMI and Agile Alliance (2017) IT17.Inthecurrentproject, theteam, the product owner, and team facilita- tor work excellently together to plan the iteration # Dybå and Dingsøyr (2008); Noll et al. (2017) # Represents original items TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Table 6. Items used to measure the Leadership dimension Items Source IT18. The team facilitator gives me helpful feedback on how to be more effective Sharma and Gupta (2012) IT19. The team facilitator eliminates barriers, encourages, and facilitates the use of agile methods # Noll et al. (2017); Senapathi and Srinivasan (2013) IT20. The team facilitator listens to my ideas and concerns Sharma and Gupta (2012) IT21. The team facilitator discusses the problems of the team Açıkgöz and Ö. İl- han (2015) IT22. The team facilitator protects the team from outside interference Ancona and Cald- well (1992) IT23. The team facilitator helps my team to acknowledge and solve our disagreements Stone and Bailey (2007) IT24. The team facilitator assists to understand whether the iteration ob- jectives are clear and whether the team agrees with these objectives # Ganesh and Gupta (2006); PMI and Agile Alliance (2017) IT25. The team facilitator gives the team helpful feedback on how to be more agile # PMI and Agile Alliance (2017); Sharma and Gupta (2012) IT26. The team facilitator is always free to support the team when busi- ness requirements conflict with the technical reality # Noll et al. (2017); PMI and Agile Al- liance (2017) IT27. The team facilitator investi- gates and helps the team to be more effective, taking into account the team velocity and the team capacity # Chagas et al. (2015); Miller (2020); Noll et al. (2017) # Represents original items Table 7. Items used to measure the Autonomy dimension Items Source IT28. In the current project, I am free to choose the tasks I want to execute in the iteraction # Karhatsu et al. (2010) IT29. In the current project, the team facilitator protects the team auton- omy from external interferences # Karhatsu et al. (2010); Moe and Dingsøyr (2008) IT30. In this organization, we have the autonomy to suggest change the team’s software process develop- ment # Patterson et al. (2005) IT31. In this team, we switch assign- ments in tasks to avoid specializa- tion and individualism # Moe and Dingsøyr (2008); Chagas (2015) IT32. The team has autonomy to adopt technical solutions without consulting the product owner or the management # Patterson et al. (2005) IT33. My team has autonomy to communicate with the product owner and other relevant stakehold- ers # Moe and Dingsøyr (2008); Chagas (2015) IT34. My team has decision author- ity and responsibility to plan the iter- ation # Karhatsu et al. (2010); PMI and Agile Alliance (2017) # Represents original items Table 8. Items used to measure the Decision-Making dimension Items Source IT35. My team has time to plan the changes without excessive stress or pressure # Jia et al. (2016); Kettunen (2014) IT36. In my team, members must NOT need to think equally # Chagas (2015); McAvoy and Butler (2007) IT37. In the iteration planning, the team analyzes the technical alterna- tives and chooses the most appropri- ate one # Chagas (2015); Moe et al. (2009); PMI and Agile Alliance (2017) IT38. In the retrospective, the team identifies, analyzes and selects im- provement items # Jia et al. (2016); PMI and Agile Al- liance (2017) IT39. My team has open and effec- tive communication # Misra et al. (2009) IT40. This organization allows the team to make their own technical decisions about the best way to de- velop the project # Patterson et al. (2005); Chagas (2015) IT41. The dependencies between the tasks do NOT hinder the fluidity of the project and do NOT cause major restrictions # Jia et al. (2016); PMI and Agile Al- liance (2017) IT42. In the current project, my work is recognized by management # Jia et al. (2016) # Represents original items TACT: An insTrument to Assess the organizational ClimaTe of agile teams - A Preliminary Study Dutra et al. 2022 Table 9. Items used to measure the Client Involvement dimension Items Source IT43. During the demo review, the team shows and validates the new functionalities with the right people # Ancona and Caldwell (1992); PMI and Agile Alliance (2017) IT44. In the current project, there are frequent meetings with business rep- resentatives and the team Serrador et al. (2018); Zaineb et al. (2012) IT45. Stakeholders always have the opportunity to suggest changes or improvements to the software # PMI and Agile Al- liance (2017) IT46. In the demo review, project problems and improvements are identified with stakeholders partici- pation # Serrador et al. (2018); PMI and Agile Alliance (2017) IT47. The current project does NOT have frequent requirement changes due to bad user stories definition # Sharma and Gupta (2012); Ahmed et al. (2017) IT48. The current project has met or exceeded the client expectations # Misra et al. (2009); Ahmed et al. (2017) IT49. The product owner is always available to explain the user stories’ details # Hoda et al. (2010); PMI and Agile Alliance (2017) # Represents original items Introduction Background Specific Characteristics for the Formation of the Organizational Climate of Agile Teams Organizational climate in agile teams TACT Overview Conceptual definition of the construct Design/adaptation/selection of items Evaluation by specialists Pretesting Case Study Planning and Execution Research questions Description of the organization and teams Data collection Case Study Results How is the organizational climate in the examined agile teams? (RQ1) Analysis of Organizational Climate from Team A Analysis of Organizational Climate from Team B Analysis of Organizational Climate from Team C How did working from home affect the organizational climate of the teams for the analyzed dimensions? (RQ1.1) How do leaders perceive TACT? (RQ2) Which are the most influential items in each dimension for the analyzed case? (RQ3) Discussion Case Study Preliminary evaluation of TACT TACT use recommendations Limitations and Threats to Validity Final Considerations Future works Appendix Constructs The items of TACT by dimension