Paper title (Paper Title style) Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 1 When Agility Meets a Project Portfolio: A Study of Success Factors in Large Organisations Masoud Aghajani Edith Cowan University, Perth, Australia Kamrul Ahsan RMIT University, Melbourne, Australia kamrul.ahsan@rmit.edu.au Naomi Whiteside Department of Education, Victoria, Australia Abstract The iterative nature of agile methods combined with high levels of team and customer interactions and continuously changing IT and software development project requirements make the management of agile project portfolios very complex. To date, the mechanisms under which project portfolio management adapts to these complexities and achieves portfolio success have not been thoroughly investigated. This study explores the notion of success and its impacting factors in large organisations' portfolios of agile IT and software development projects. Using a multiple case study design, we analysed the agile project portfolios of seven large organisations. We identified four success criteria and 15 success factors and categorised them into a unique agile portfolio success framework. Some of these criteria and factors are unique to agile project portfolios. The framework contributes to agile and project management literature by conceptualising the notion of success in portfolios of agile projects while revealing a set of factors that affect the relationship between an agile portfolio with its subcomponents and the surrounding environment. The framework supports managers and practitioners in large organisations in reflecting on their agility efforts to achieve higher success rates in their agile portfolios. Keywords: Agile project management, project portfolio management, agile portfolio, IT and software development projects, success factor. 1 Introduction Agile project management (APM) has emerged as an approach for project planning and delivery that addresses many of the shortcomings of traditional IT and software development (ISD) projects (such as waterfall projects) and improves project results (Gemino et al., 2021; Serrador & Pinto, 2015). Unlike the traditional project management approaches that emphasise upfront planning and sequential project delivery, APM methods rely on iterative planning and delivery with a focus on incremental product delivery through fast and flexible interactions between self-managed development teams and customers (Schwaber, 2004). Through their collective processes, routines and relationships with their environment, agile methods create and embrace change to achieve customer value (Conboy, 2009). Coordinating agile project delivery is primarily done through routine activities such as daily stand-up meetings, iterative product reviews and team retrospectives (Schwaber, 2004; Thummadi et al., 2011). Recent research indicates the widespread adoption of agile methods in small, large, and multiproject ISD project environments (Digital.ai, 2021; Jørgensen, 2019; Paasivaara et al., 2018). Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 2 While there is an extensive body of research on small agile projects and teams, little research has examined the impacts of agile methods on the management of project portfolios (Stettina & Hörz, 2015; Sweetman & Conboy, 2018). Project portfolio management (PPM) entails balancing shared resources and risks between multiple projects to maximise business value and achieve strategic alignment (Cooper et al., 1999; Meskendahl, 2010). PPM is perceived as a governance structure with a critical management decision-making role in aligning projects with organisational strategy and achieving business value through effective resource allocation and risk management (Mosavi, 2014). PPM is still an emerging aspect of strategic business management that aims to improve the chances of project success (Hansen & Svejvig, 2022; Padovani & Carvalho, 2016). With PPM, organisations can achieve greater project visibility and higher product development efficiency, possibly leading to competitive advantage and greater chances of business success (Killen et al., 2008a; Martinsuo, 2013). Roughly 60% of organisations utilise portfolio structures to manage their projects (KPMG, 2017). PPM and APM intersect in ISD project portfolios. However, managing multiple agile projects under project portfolios is particularly challenging and troublesome (Conboy & Carroll, 2019; Stettina & Hörz, 2015; Sweetman & Conboy, 2018). Agile projects result in a high degree of complexity at the portfolio level because the evolving and flexible nature of agile projects, combined with the self-organising and autonomous setup of agile teams, are at odds with traditional PPM's well-established and formal organisational processes (Dingsøyr & Moe, 2014). When aggregated at the portfolio level, the intensified interactions of agile teams and customers can create unmanageable tensions and complexities that threaten the success of agile efforts and PPM (Sweetman & Conboy, 2018). Sweetman and Conboy (2018) indicate that portfolio governance, in traditional PPM, is stable over time as projects are usually plan-driven and predictable. However, projects of an agile portfolio are pushed in different directions with continuously changing project requirements and customer demands. Consequently, even though agile projects may be successful individually, their portfolio governance can become disjointed, incoherent and complex if not managed effectively. Scholars call for more research to adapt PPM to the tensions and complexities of multiproject agile environments (Stettina & Hörz, 2015; Sweetman & Conboy, 2018). Despite the rapid adoption of agile methods at scale and in portfolios of ISD projects, very little is known about the notion of success in agile portfolios. Prior agile literature has primarily focused on individual agile project success (Chow & Cao, 2008; Misra et al., 2009; Stankovic et al., 2013) and has identified Success Factors (SFs). SFs refer to areas that require continuous and careful attention to achieve organisational goals (Fortune & White, 2006). However, the identified individual agile project SFs do not address the complexities of managing multiple agile projects under a portfolio. There is also a wealth of research on factors influencing traditional1 portfolio success (Kaufmann et al., 2020; Kock & Gemünden, 2020; Kopmann et al., 2017). However, there has been limited attention to multiproject agile environments as a context for PPM. Therefore, we respond to recent calls for research on the consequence of APM on PPM effectiveness and success (Niederman et al., 2018; Sweetman & Conboy, 2018) by aiming to explore the notion of success for agile project portfolios by investigating their success criteria (SCs) and SFs in large organisations. In this study, portfolios of agile projects (or agile 1 In contrast to agile portfolios, non-agile portfolios are referred to as 'traditional' portfolios. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 3 portfolios) refer to the integrated management of multiple agile ISD projects under independent portfolios. The study focuses on large organisations because they are more likely to practice PPM to deal with multiple agile ISD projects. We adopted the definition of large organisations used by the Organisation for Economic Cooperation and Development (OECD), which defines large as having at least 250 employees (OECD, 2021). Therefore, this study seeks to address the following questions: RQ1: What constitutes success in portfolios of agile projects in large organisations? RQ2: What factors contribute to the success of portfolios of agile projects in large organisations? The present research contributes to understanding agile ISD portfolios and the conditions under which these portfolios achieve success. We use multiple case study research to develop an agile portfolio success framework with agile portfolio SCs and SFs as its subcomponents. The framework can potentially guide large organisations in adapting and sustaining agile methods at scale and their agile ISD portfolios. The following section reviews the relevant success literature in agile scale-up and traditional and agile PPM while also revealing the theoretical underpinning of this study. Section 3 describes the multiple case study research design and presents seven case organisations. Section 4 unveils the research data analysis and findings, and then Section 5 discusses the key findings in line with previous research and explains the research implications and limitations. The last section summarises the findings and suggests future research directions. 2 Literature Review We review the relevant literature across two main domains, agile scale-up and project portfolios, to identify SFs in existing research, reveal gaps/problems in previous research and introduce a theoretical lens. 2.1 Agile Scale-up: Challenges and Success Factors Rigby et al. (2018) set the ultimate goal of scaling agile in a function like ISD to transition from several small agile innovation teams to many more teams with the vision to make agile the dominant way the organisation operates. Scaling agile brings agile values and principles to business operations and support functions to achieve greater flexibility and efficiency (Ebert & Paasivaara, 2017). Dingsøyr and Moe (2014) argue that scaling agile involves two aspects. The first aspect is increasing the number of agile teams involved in higher levels of decision- making and planning by using agile practices (e.g., release planning and road mapping). Second, scaling the system engineering activities executed in each sprint to a truly iterative practice instead of a stage-gated planned approach that supports cross-functional teams. When scaling agile in a large organisation, multiple teams may collaborate to develop various subsystems of a single product or to produce different products that are part of a product family (Beecham et al., 2021). Dingsøyr et al. (2017) point out that scaling agile involves upward integration of agile principles and practices in the organisational hierarchy, from teams to programs and portfolios, and higher to the executive suite. Several agile scale-up frameworks have been proposed by agile practitioners to guide organisations through the scale-up process. According to the 15th state of agile report by Digital.ai (2021), Scaled Agile Framework (SAFe) (Leffingwell, 2018) and Scrum@Scale Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 4 (Sutherland, 2021), with 37% and 9% adoption rates are the most dominant agile scale-up frameworks. Enterprise Scrum (ES) (Herman, 2017), Spotify model (Kniberg & Ivarsson, 2012), Agile Portfolio Management (AgilePfM) (Agile-Business, 2017), Disciplined Agile Delivery (DAD) (Ambler & Lines, 2012) and Large Scale Scrum (LeSS) (Larman & Vodde, 2013) are also used but with much lower adoption rates. These frameworks comprise various practices and processes supported by predefined roles, artefacts and rituals. Among these frameworks, only SAFe and AgilePfM clearly recognise the need for PPM. The full version of SAFe (Version 5) consists of three layers: portfolio, large solution and essential. The portfolio level aligns organisational strategy with lower-level solution development by defining several portfolio- level roles, practices and artefacts (Scaled-Agile, 2021). However, the framework is criticised as being highly prescriptive and inflexible for ISD portfolios (Geraghty, 2020; Schwaber, 2014). AgilePfM focuses on continuous value delivery and introduces the concept of rolling-wave planning for the dynamic management of agile portfolios using common agile practices such as stand-ups and retrospectives (Dingsøyr et al., 2019). In general, while each of these frameworks has its own advocates and critics, there is limited empirical evidence on the effectiveness and success of these frameworks (Beecham et al., 2021; Conboy & Carroll, 2019). For agile scale-up, the systematic literature review by Dikert et al. (2016) identified 29 SFs from previous literature and categorised them under 11 categories. The important SF categories were choosing and customising the agile approach, top management support, focusing on agile values, ensuring management support and providing training and coaching. The action research by Kalenda et al. (2018) explored a software company going through large-scale agile adoption. They confirmed a few previously identified SFs (i.e., management support and focusing on agile values) while revealing two new SFs: appropriate organisational culture and prior agile team experience. However, the authors did not clearly elaborate on the cultural elements essential for a successful agile scale-up initiative. Rigby et al. (2018), drawing on the authors' past experiences, provided recommendations on building and sustaining agile at scale, such as getting agile rolling with a small wave of agile teams (piloting), sequencing the adoption process and acknowledging the importance of acquiring talented and motivated agile practitioners. A recent systematic literature review by Edison et al. (2021) identified 27 SFs for agile scale-up frameworks and organised them into four categories: management and organisational, process, people and technology. Authors found it challenging to validate each agile scale-up SFs with different agile scale-up frameworks because of the association of these SFs with studies with different levels of analysis and granularity. They also found that not all agile scale-up frameworks (such as DAD and Scrum@Scale) received adequate attention from scholars. Table 1 summarises the previously identified SFs for agile scale-up that were discussed in this section. Overall, the empirical findings on agile scale-up and its success research are still in the early stages, and evidence of sustaining agile scale-up frameworks is still scarce (Conboy & Carroll, 2019). Furthermore, the dominance of a high number of grey literature on agile studies could indicate agile practitioners’ interest and concerns about agile scale-up practices and frameworks. However, while the number of empirical studies addressing agile scale-up frameworks is on the rise (Edison et al., 2021), academia still lags behind the practice in this domain. Moreover, while some agile scale-up SFs are reported in recent studies, these SFs are mainly linked to the adoption stages of agile at scale or a specific agile scale-up framework and do not cover the post-adoption stages of agile transformation. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 5 Agile scale-up SFs Brief description Supporting references Top management support Top management decisions and support are critical for the success of agile scale-up efforts. Successful agile scale-up transition requires strong and aligned management. (Dikert et al., 2016; Edison et al., 2021; Kalenda et al., 2018; Rigby et al., 2018; Shameem et al., 2017) Communicate agile values The agile way of working will take root with intensive and transparent communication. Moving towards agile scale-up is assisted by making agile benefits and values visible. (Dikert et al., 2016; Kalenda et al., 2018; Rigby et al., 2018) Customising agile methods Organisations should customise their agile approach and practices to fit their unique challenges and needs. (Conboy & Carroll, 2019; Dikert et al., 2016; Edison et al., 2021) Piloting Successful piloting clears disbelief about the appropriateness of the agile scale-up method and serves as a learning experience that enhances the adoption process. (Dikert et al., 2016; Kalenda et al., 2018; Paasivaara et al., 2018) Training and coaching Training on methods and practices improve the chances of succeeding in scaling up agile. Agile practices are best learned by doing, and coaching can help sustain agile practices and mindset. (Conboy & Carroll, 2019; Dikert et al., 2016; Edison et al., 2021; Shameem et al., 2017) Engaging people Engaging stakeholders allows an organisation to gain acceptance and legitimise the agile scale-up process. Inclusion will motivate people to participate and work in a new agile way. (Bass & Haxby, 2019; Dikert et al., 2016; Edison et al., 2021; Kalenda et al., 2018; Kasauli et al., 2021) Value agile communities Establishing agile communities influences the organisational culture and allows the formation of agile skills among teams. (Dikert et al., 2016; Edison et al., 2021; Shameem et al., 2017; Smite et al., 2019) Allow teams to self- organise Self-organisation creates commitment to the change and allows teams to take ownership of product development. Such empowerment increases teams' productivity and morale. (Dikert et al., 2016; Edison et al., 2021; Shameem et al., 2017) Requirements management Many requirement engineering challenges are reported in the agile literature. Organisations require to invest in and implement effective requirement discovery practices. (Bjarnason et al., 2022; Dikert et al., 2016; Kalenda et al., 2018; Kasauli et al., 2021; Shameem et al., 2017) Align the organisation Organisational goals should be set and aligned through more fit-for-purpose, flatter and flexible processes. Agile roles need to be defined initially to ensure coverage of strategic priorities. (Conboy & Carroll, 2019; Dikert et al., 2016; Edison et al., 2021) Appropriate organisational culture An appropriate organisational culture (e.g., trust culture) fosters collaboration among teams and other stakeholders. (Abrar et al., 2021; Edison et al., 2021; Kalenda et al., 2018) Experienced and disciplined teams A high level of agile team members' technical and social knowledge and expertise is critical to agile scale-up success. (Edison et al., 2021; Kalenda et al., 2018; Rigby et al., 2018; Shameem et al., 2017; Šmite et al., 2017) Shared vision A shared vision assists an organisation in setting a common ground for all stakeholders and clarifies organisational strategies and project goals. (Edison et al., 2021; Kalenda et al., 2018; Shameem et al., 2017) Information and knowledge sharing An appropriate knowledge-sharing system supports communication, decision-making, knowledge networks, and social capital through the agile scale-up process. (Dingsøyr et al., 2019; Edison et al., 2021; Kalenda et al., 2018; Shameem et al., 2017) Table 1. The summary of agile scale-up SFs from the literature Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 6 Overall, the empirical findings on agile scale-up and its success research are still in the early stages, and evidence of sustaining agile scale-up frameworks is still scarce (Conboy & Carroll, 2019). Furthermore, the dominance of a high number of grey literature on agile studies could indicate agile practitioners’ interest and concerns about agile scale-up practices and frameworks. However, while the number of empirical studies addressing agile scale-up frameworks is on the rise (Edison et al., 2021), academia still lags behind the practice in this domain. Moreover, while some agile scale-up SFs are reported in recent studies, these SFs are mainly linked to the adoption stages of agile at scale or a specific agile scale-up framework and do not cover the post-adoption stages of agile transformation. 2.2 Traditional Portfolios: Success Criteria and Success Factors Traditional PPM is an established research field, and numerous studies have contributed to its advancement by identifying its characteristics, methods, processes and goals (Hansen & Svejvig, 2022). In traditional PPM, portfolio SCs are defined as achieving well-established PPM goals, i.e., achieving average success over all projects, using synergies among projects, finding strategic alignment and portfolio balancing (Cooper et al., 1999, 2002; Meskendahl, 2010). Achieving average success over all projects within a portfolio is linked to delivering the projects on time, within budget and to specifications extended by meeting customer satisfaction dimensions of project success (Meskendahl, 2010; Shenhar et al., 2001). Synergy exploitation is about achieving greater benefits through the integrated management of multiple projects. These benefits can include enhancing technologies, markets, knowledge and resource synergies. For instance, coordinating the interdependencies among projects that use the same technology or operate in the same market may increase efficiency or shared opportunities, which may not be achievable through the independent management of each project (Kopmann et al., 2017; Meskendahl, 2010). According to Voss and Kock (2013), increased project portfolio interdependency necessitates more coordination but can also lead to better use of synergies. Strategic alignment can be achieved when projects under a portfolio fit the organisation’s strategy (Hoffmann et al., 2020; Kopmann et al., 2017). Portfolio balancing refers to attaining an equilibrium of risks, resources and long-term versus short-term opportunities when executing a portfolio (Killen et al., 2008b; Teller et al., 2012). Previous research has identified various SFs for traditional PPM. Earlier contributions of PPM scholars have focused on factors related to PPM structure, for example, establishing formal portfolio processes and top management involvement and support. Project management- related factors as a subset of PPM are also explored. For instance, information availability, sharing and quality (Jonas et al., 2013; Martinsuo & Lehtonen, 2007; Unger et al., 2012) and project goal setting (Biedenbach & Müller, 2012; Martinsuo & Lehtonen, 2007) are proposed as project-level SFs contributing to portfolio success. Recent research on PPM has focused on identifying organisational capabilities as SFs. For example, Kock and Gemünden (2020) showed that portfolio innovativeness as a dimension of entrepreneurial orientation moderates the relationship between strategic PPM practices (portfolio structuring and steering) and portfolio success. Kaufmann et al. (2020) showed the positive influence of two antecedents of organisational agility (i.e., entrepreneurial orientation and voice behaviour) on strategy recognition and further portfolio success. Voice behaviour in an organisation relates to the cultural support an individual receives to willingly and constructively express concerns and opportunities. Kopmann et al. (2017) found a positive and significant relationship between portfolio strategic control practices and portfolio success, Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 7 while both deliberate strategy recognition and emerging strategy recognition mediated the relationship. With deliberate strategy recognition, they referred to the process of purposefully cascading the formulated corporate strategy from higher levels of the organisation’s hierarchy to the project level, while emerging strategy recognition was about disclosing emerging patterns in a portfolio to inform the strategy formulation process. In another major study, Kock et al. (2020) showed the overall positive effect of PPM information systems (PPMIS) on the quality of PPM processes and portfolio success. One interesting finding in their study was that the PPMIS effect could only materialise when highly formal project management, PPM and risk management practices existed. Table 2 presents a broad selection of traditional PPM SFs. Traditional PPM SFs Brief description Supporting references Establishing formal portfolio processes Formalisation of projects and portfolios through configuring standardised routines and processes positively impacts portfolio quality, facilitates resource allocation and improves transparency in projects and portfolio environments. (Blomquist & Müller, 2006; Jonas, 2010; Kester et al., 2014; Kock et al., 2016, 2020; Kopmann et al., 2015; Teller, 2013; Teller et al., 2012; Teller & Kock, 2013) Information availability, sharing and quality Information availability for decision-makers is one significant project-level factor contributing to portfolio success. Also, a lack of information quality, sharing and transparency is a significant barrier to sound decision-making at the portfolio level. (Dietrich & Lehtonen, 2005; Jonas, 2010; Jonas et al., 2013; Lerch & Spieth, 2013; Martinsuo & Lehtonen, 2007; Spieth & Lerch, 2014; Teller et al., 2012) Top management involvement and support Senior managers’ involvement in portfolio decision-making (e.g., advocating to implement a strategy or re-evaluate/terminate projects) is identified as an essential contributor to portfolio success. Top managers’ unsupportive behaviour or unintentional enforcement of undesired projects negatively impacts portfolio success. (Beringer et al., 2013; Blomquist & Müller, 2006; Jonas, 2010; Jonas et al., 2013; Kock et al., 2015; Rank et al., 2015; Unger et al., 2012) Project goal setting Appropriate project goal setting improves project management efficiency and contributes to portfolio success. Project goal setting should be expanded towards broader business goals if better portfolio results are expected. (Biedenbach & Müller, 2012; Martinsuo, 2013; Martinsuo & Lehtonen, 2007) Internal stakeholder engagement (e.g., project managers, portfolio managers and line managers) Previous research indicates the significant influence of line managers, project portfolio managers and project managers on portfolio success. For example, the project manager's influence, resulting from their authority, responsibility and engagement towards project outcomes, plays a crucial role in the success of project portfolios. (Beringer et al., 2013; Blomquist & Müller, 2006; Jonas, 2010; Petro & Gardiner, 2015; Unger et al., 2012) Strategic PPM practices Portfolio practices such as portfolio structuring (e.g., stakeholder management and strategic planning), portfolio steering (e.g., business case monitoring, strategic control and adaptiveness) and risk management (risk processes and culture) are positively related to portfolio success. (Dietrich & Lehtonen, 2005; Jonas, 2010; Kaufmann et al., 2020; Kock & Gemünden, 2020; Kopmann et al., 2017; Müller et al., 2008; Teller, 2013; Teller et al., 2014; Teller & Kock, 2013) Entrepreneurial orientation Entrepreneurial orientation refers to “the strategy-making processes that key decision makers use to enact their firm’s organisational purpose, sustain its vision, and create competitive advantage(s)” (Rauch et al., 2009, p. (Kaufmann et al., 2020, 2021; Kock & Gemünden, 2020; Rank et al., 2015) Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 8 Traditional PPM SFs Brief description Supporting references 763). Dimensions of EO (e.g., portfolio innovativeness) are essential factors for PPM performance. Customer relationship value Both relationship values for and from the customer (such as monetary terms, product ideas, access to new markets, or other virtues) independently contribute to portfolio success. Such relationship values are even more critical in complex portfolios. (Voss, 2012; Voss & Kock, 2013) PPM absorptive and adaptive capabilities The early utilisation of absorptive capabilities (i.e., the firm’s ability to utilise external knowledge through various ways of learning) is essential for project and portfolio performance. Also, adaptive capabilities (i.e., the capability to identify and capitalise on emerging market opportunities, e.g., global marketing monitoring and market/technology sensing) positively impact portfolio success. (Biedenbach & Müller, 2012; Killen et al., 2012) PPMIS PPMIS application positively and significantly relates to PPM quality, promoting portfolio success. PPMIS application positively impacts project and portfolio formalisation. (Kock et al., 2020) Table 2. The summary of portfolio SFs from the literature Most portfolio success studies view traditional PPM as a rational and plan-driven decision- making process. While this viewpoint has contributed to improving organisations’ PPM structure and performance (Martinsuo, 2013), it is limited in three ways. First, portfolio decision-making is less rational when its context (the unique conditions under which the portfolio is being managed) is considered. For example, the projects and portfolio selection can become politically motivated and path-dependent instead of being highly rational and planned (Martinsuo, 2013). Also, the day-to-day practice of PPM may deviate from its intended rational and plan-driven approach, as portfolio managers often have to apply it in uncertain and dynamic situations that they cannot fully comprehend. Second, portfolio success draws on what the organisation and its shareholders value. However, the perception of value may differ depending on the stakeholders’ priorities, knowledge and attitudes (Shenhar et al., 2001). Stakeholders may hold conflicting values and different understandings about portfolio success (Beringer et al., 2013; Martinsuo & Killen, 2014). Also, internal and external stakeholders can potentially promote, support, slow down, or limit the PPM process, and their involvement is crucial for sustaining the project portfolios (Beringer et al., 2013; Sánchez, 2015). However, current traditional PPM research does not clearly show the impact of different/conflicting viewpoints of stakeholders on PPM. Third, to our knowledge, very few studies acknowledge the significant relationship between multiproject customers and portfolio success (Voss, 2012; Voss & Kock, 2013). Current PPM research largely neglects the consequences of customer involvement, competencies, and actions on portfolio success. 2.3 Agile Portfolios: The Research Problem Research that focuses exclusively on agile portfolios has been minimal. The early attempts to explore these portfolios are mostly experience reports explaining transitioning from traditional to agile portfolios and identifying challenges, benefits and practices that Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 9 accompany said transition (Laanti & Kangas, 2015; Rautiainen et al., 2011; Thomas & Baker, 2008). Few theoretical frameworks are developed for agile portfolios (Krebs, 2008; Vahaniitty, 2012), although we found no evidence of their use by practitioners. Stettina and Hörz’s (2015) multiple case study is perhaps one of the first attempts to empirically explore portfolios of agile projects. Most of the 14 cases in their study undertook a bottom-up approach to adopt and scale agile methods. For instance, organisations first adopted agile methods at the project level and then expanded their agility efforts toward PPM as the next neighbouring domain of practice. Their study revealed several important challenges in managing agile portfolios, e.g., alignment to existing processes, lack of involvement and commitment on the side of senior managers and difficulties in resource alignment (Stettina & Hörz, 2015). Using complex adaptive systems theory and expert interviews, Sweetman and Conboy (2018) developed propositions to address how PPM can adapt to the tensions that arise from multiple agile projects under a portfolio. One tension was the increased number of teams and customer interactions in APM that can become unmanageable and cumbersome when aggregated at the portfolio level. Also, the iterative and evolving nature of APM combined with the self- organising and autonomous routines of agile teams can cause continuous challenges to the portfolio mix of agile projects. These agile characteristics are particularly at odds with PPM's centralised, formal and plan-driven structure (Stettina & Hörz, 2015; Sweetman & Conboy, 2018). In such conditions, whilst a portfolio can stabilise through governance and oversight, its agile projects’ capability to address change might diminish, resulting in APM losing its significance (Sweetman & Conboy, 2018). Therefore, Sweetman and Conboy (2018) propose that portfolio managers should find the right balance between control and autonomy in their agile projects and portfolios. Hoffmann et al. (2020) employed activity theory to investigate a single revelatory case in the context of agile IT PPM. They empirically derived three design goals (alignment, efficiency and agility) and nine supporting design principles for agile IT PPM. The principles are continuous strategic alignment, short cycle portfolio planning, termination of non-value-adding projects, project approval only at the last moment, approval of only short projects, establishing a portfolio buffer, preventing resource overload, avoiding multitasking and ensuring uninterrupted project completion. However, in their study, efficiency was limited to meeting business requirements, quality, target dates and budget. Hoffmann et al. (2020) and Sweetman and Conboy (2018) concluded that empirical studies are needed to better understand what constitutes success in agile portfolios. 2.4 The Theoretical Lens and Conceptual Framework Informed by recent research on the relationship of project portfolio with its context (Martinsuo, 2013; Martinsuo & Geraldi, 2020), we used contingency theory (CT) as a theoretical lens to develop a conceptual framework that guided the process of qualitative data collection and analysis. CT asserts that relationships between organisational characteristics (e.g., structure, strategy and management practices) and its performance depend on the organisation's context (Donaldson, 2001). Accordingly, no single form of organising can be suitable for every contextual setting, so a fit between the context and organisation is required (Shenhar, 2001). In project management, CT portrays projects as “temporary organisations” within their parent organisations and indicates that the universal “one size fits all” approach to managing projects may be inadequate (Shenhar, 2001, p. 395). Also, the current view of PPM frames a project portfolio as an organisation hosting multiple projects and programs as temporary organisations (Bechtel et al., 2021; Hansen & Svejvig, 2022; Martinsuo & Geraldi, 2020). Such a Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 10 view opens up new perspectives on how project portfolios are organised, performed and evolved in relation to their internal and external contingencies (Hansen & Svejvig, 2022; Martinsuo & Geraldi, 2020). A contingency is “any variable that moderates the effect of an organisational characteristic on organisational performance” (Donaldson, 2001, p. 5). Following Bechtel et al. (2021) and Martinsuo and Geraldi (2020), we conceptualise a project portfolio as a semi-temporary organisation with limited budgets and time constraints that function as a link between permanent (parent organisation) and temporary (programs and projects) forms of organising. Martinsuo and Geraldi (2020) argue that as project portfolios are typically concerned with product and service development projects, their success is critical in achieving overall business success and the firm’s market positioning. The project portfolio, as the collection of multiple projects, constitutes the portfolio subunits and the environment surrounding the portfolio (Bechtel et al., 2021). Therefore, we argue that for an ISD portfolio to succeed as an organisation, its characteristics should fit between its subunits (agile projects) and its organisational environment (context). Hence, in Figure 1(a), we depict how this study considers agile projects and their related ISD portfolio in an organisational setting, which will explain the interactions between different levels (i.e., agile projects, portfolio and organisational environment). Previous research has explored various dimensions of traditional portfolio success (Cooper et al., 1999, 2002; Jonas, 2010; Meskendahl, 2010), while the effect of project-level contingencies on portfolio success is also considered. For example, Teller et al. (2012) showed the association of single project management formalisation (configuration of standardised routines and practices) with increased PPM quality and portfolio success; and Martinsuo and Lehtonen (2007) found projects’ information availability and goal setting as significant contributors to portfolio success. Also, many studies have tested the effect of contextual contingency variables on the link between certain management aspects and traditional portfolio success (Kaufmann et al., 2021; Kock & Gemünden, 2020; Voss & Kock, 2013). However, agile projects differ significantly from traditional plan-based projects as they comprise different routines, processes and artefacts (Stettina & Hörz, 2015; Thummadi et al., 2011). Further, agile projects continuously evolve in response to frequently changing project requirements and customer demands, making them quite different from traditional plan- based projects (Sweetman & Conboy, 2018). Therefore, we focus on understanding portfolio success in agile project environments by identifying its SCs and SFs in large organisations. To analyse the portfolio success with theoretical rigour, considering CT as a theoretical lens, we developed a conceptual framework shown in Figure 1(b). We distinguish between agile portfolio SFs by categorising them at two levels: project-level SFs (P-SFs) and contextual SFs (C-SFs). We consider P-SFs as contingencies related to individual agile projects and teams as portfolio subunits that may influence agile portfolio SCs. We also investigate SFs that characterise the agile portfolio in the environment (context) under which the agile portfolio is operating. These C-SFs comprise contingency factors related to the agile portfolio environment, which may be related to the organisational structure and culture affecting portfolio success. Following the influence of CT on the organisational context, we further explored an extension of CT called the Uncertainty/Consequences framework (UC framework) (Howell et al., 2010) to interpret the research results. The UC framework classifies project (and portfolio as a collection of projects) contingency factors into two types, uncertainty (U) and consequences (C). It considers these categories as two orthogonal dimensions determining the appropriate Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 11 process model (i.e., agile or plan-driven) for managing a particular project or portfolio (Howell et al., 2010). We argue that each identified portfolio SF (P-SFs or C-SFs) can fall into either U or C categories. Therefore, we use U and C dimensions to cluster the identified agile portfolio SFs in Section 4. In what follows, we explain the U and C dimensions from the literature. The U dimension represents the probability that something unexpected will happen in the project and includes contingencies related to uncertainty, complexity and urgency resulting from a single project or portfolio environment, scope and outcomes. Uncertainty includes probabilistic outcomes and the ambiguity of situational parameters related to goals, methods, markets, technology or external stimuli (Howell et al., 2010; Williams, 2005). Complexity is defined as the degree of differentiation and interdependence of project or portfolio components related to scope, environment, diversity and the size of the parent organisation, portfolio, project or team (Howell et al., 2010; Kock et al., 2016). The challenge of complexity lies in comprehending and making sense of the intricacies involved, which ultimately affects the relationship between inputs and outputs of a project (or portfolio). A lack of predictability is synonymous with uncertainty, and thereby complexity becomes a factor in uncertainty (Howell et al., 2010). Urgency is the extent to which time constraints are a limiting factor (of limited information) in a single project or portfolio activities and decision-making. In projects and portfolios, urgency limits the ability to make decisions because of limited information. Managers under time pressure tend to take more control of the situation, often by short- sighted measures. These effects increase the probability of unexpected project behaviour (Howell et al., 2010; Williams, 2005). Figure 1. The research setting and conceptual framework The C dimension represents how much it will matter if some uncertainties occur. The C factors are related to criticality and team empowerment. Criticality can be seen as how much is at stake in a single project or portfolio. In other words, it is the impact or consequences of unexpected or unmanaged events (e.g., project failure) on the project/portfolio goals and, consequently, the organisation. Empowerment-related factors such as authority levels, team Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 12 size and geographic distribution are often the consequences of organisational culture that the parent organisation frequently imposes. Changing such parameters may be infeasible for projects and portfolios. The ability to make such changes is an empowerment issue because it may or may not be authorised by the parent organisation. In a project or portfolio organisation, the consequences of unexpected events depend heavily on how the team manages them. Team empowerment factors affect the team’s ability to communicate, comprehend and quickly respond to unexpected events in projects and portfolios (Howell et al., 2010). Hence, it could improve the chances of dealing with unexpected events and increase the chances of project and portfolio success. Both the U and C dimensions can explain if a single project or portfolio can be managed using an agile or plan-driven process model. For example, in an agile project, if the dominant issue is understanding its ill-defined goals at the initial stages, the team must adopt an iterative process involving partial delivery of the goals and use feedback to continuously redefine the goals to reduce unexpected consequences. In that case, the project success is independent of U. The approach assumes high uncertainty; if, in practice, it is lower, this has no effect on success. However, the agile project’s success is limited to the C dimension. Any event that the project team cannot handle may lead to failure. Also, the team will not necessarily know if they have effectively handled the event until later stages of the project. Conversely, suppose the successful execution of a project relies on the optimal sequencing of available resources and constraints in a project plan rather than the project objectives. In that case, the success of the chosen process model is independent of C because the objective is to anticipate and avoid changes, and their consequences are less irrelevant. Hence, the performance of the process model will vary by how effectively project uncertainty is addressed. The UC framework suggests traditional plan-driven approaches to project delivery under such conditions. The developed conceptual framework (with U and C dimensions) helped to categorise the identified SFs to form the study’s research framework, as is explicated in Section 4. The next section explains the research process and methodology. 3 Research Methodology This research explores the notion of success and its related SFs in portfolios of agile ISD projects in large organisations. We used a qualitative multiple case study design to investigate agile portfolios from real-life scenarios, especially when knowledge and perspective on agile portfolio SCs and SFs are limited in the literature. The case study design is suitable when contextual conditions are crucial for understanding the phenomena and the boundaries between phenomenon and context are unclear (Yin, 2018). The multiple case study design allowed us to collect multiple perspectives and experiences of participants in different organisations and industries. This section explains the case and participant selection process, data collection and analysis procedures and the steps taken to ensure process and data validation. 3.1 Case and Participant Selection Process Project portfolios are usually found in large organisations where the alignment of multiple projects and programs with business strategy is crucial (Kopmann et al., 2017). Hence, the study’s unit of analysis was limited to a portfolio of agile ISD projects in a large organisation. We exclusively selected and investigated portfolios of agile ISD projects within the studied cases because these portfolios show significantly different patterns compared to traditional Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 13 (non-agile) portfolios (Sweetman & Conboy, 2018). We used replication logic to select cases that either predict similar results or produce contrary and extending outcomes. Then, multiple cases were treated as discrete experiments that provided replications, contrasts and extensions, aiding the theoretical development of findings (Eisenhardt & Graebner, 2007; Yin, 2018). Four criteria guided the case selection process. First, the potential case organisation had to practise PPM and use APM for its ISD projects. We did not consider organisations that use APM for applications outside the ISD field. Second, the potential organisation had to have at least 250 employees to be considered large, following the business size definition by OECD (2021). Third, the selected agile portfolio must have contained at least three agile teams to be considered in this study. The criterion was derived from the taxonomy of scale proposed by Dingsøyr et al. (2014). Forth, the potential organisation should have practised APM and PPM together for at least three years. We assumed such a period would be crucial for developing an agile mindset and culture within the case organisation. Following these criteria, an initial list of 35 potential organisations operating in Australia was created. We sought potential participants with active roles in agile projects, programs and portfolios within the potential case organisations with at least three years of agile experience. Overall, 125 potential participants were identified and contacted. The eligibility of potential participants for inclusion was checked through available public information (such as participants’ LinkedIn profiles) and double-checked further upon interviews. Overall, seven case organisations and 13 participants consented and participated in the study. Table 3 contains the case descriptions, including their industry, product offerings, size, type of investigated portfolios, number of agile teams and agile methods in use and agile practice history. Table 3 also reports information about participants and their agile experience at the time of the interviews. The letters A to H symbolise the anonymised cases. Case Industry, product offering & size Type of portfolio, number of agile teams & methods in use Agile practice history Case participants & agile experiences A Education and training It provides tertiary and vocational education along with research services. It operates with over 10,000 staff and manages its projects with multiple portfolios. IT services portfolio The portfolio was structured under the operations business unit and consisted of more than six agile teams (7 members each) using SAFe, Scrum, XP, Kanban, and waterfall to deliver projects. The agile journey started in 2014 when the company experienced financial loss and delays in ISD project delivery. Senior managers hired a consultancy firm to support the transition to SAFe, delivering three major ISD programs under one portfolio using seven agile teams. The initial agile adoption resulted in increased product quality, faster value delivery and improved project visibility. Two people: Release train engineer (RTE) (7 ys) Portfolio manager (PfM) (10 ys) B Construction digital collaboration solution It provides web- based and mobile collaboration technologies for ISD portfolio The portfolio was developing a range of digital solutions for various industries. Over 25 agile teams (7 members each) were operational, using Agile was a salvation for the company after its operations dealt with increasing product complexity, unhappy customers, and growth pressures in 2009. While the developers worked tirelessly to test and fix software products, they could not release Two people: Vice president of collaboration (VP) (15+ ys) Scrum master (SM) (12+ ys) Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 14 Case Industry, product offering & size Type of portfolio, number of agile teams & methods in use Agile practice history Case participants & agile experiences projects and processes on software-as-a- service (SaaS). It operated with about 850 staff and ran its projects under one portfolio. Scrum, XP, and Kanban to deliver projects. them fast enough. Agile adoption followed a bottom-up approach that led to the complete reinvention of the company’s software delivery process. As a result, features could be added more quickly, which improved team and customer satisfaction. C Store-based retailing It provides retailing and consumer services for groceries, merchandise, and other consumer services. It operated with about 100,000 staff, and its projects ran under multiple portfolios. IT operations portfolio The portfolio was structured under the operations business unit and consisted of five agile teams (9-10 members each) using Scrum, XP, and waterfall to deliver projects. The company started agile practices in 2014. Then, agile projects experienced a halt between 2015 to 2017. The agile scale-up started in 2017 by hiring a new, highly experienced agile champion. Agile adoption was a response to increasing pressure in the competitive retail industry to deliver outcomes faster. The company had several independent agile ISD project portfolios, and we investigated one. Two people: An agile coach (AC) (10 ys) Agile delivery manager (ADM) (5 ys) D Telecom, mobile and internet It provides consumers with telecommuni- cation networks, mobile, internet and cable TV services. It had more than 32,000 staff, and its projects ran under multiple portfolios. Digital experiences portfolio The portfolio operated under the networks and IT business unit and consisted of over 25 agile teams (9-10 members each) using SAFe, Scrum, XP, Kanban, and waterfall to deliver projects. The company adopted agile practices in 2010, initiating an IT delivery program to improve software quality and speed of delivery. Later, the company hired a consultancy firm to implement SAFe in one of its portfolios. In 2015, it partnered with another consultancy firm to help train more agile staff to transform the entire business into agile. In 2017, the company reported 100 agile teams in multiple ISD project portfolios. Two people: Agile coach (AC) (5 ys) Business analyst (BA) (4 ys) E Insurance services It provides investment, insurance, superannuation, and financial services. It had over 5000 staff, and its projects ran under multiple portfolios. ISD products portfolio The portfolio operated under the innovation and transformation department and consisted of over seven agile teams (7 members or less) using Scrum for its projects. The company was acquired and then restructured by an international firm in 2016. However, the company’s technology and digital staff have been involved in agile practices since 2014. In late 2016, the company hired an agile consultancy firm to help deploy SAFe but later backed down from the decision as they found little interdependencies between their ISD projects. Two people: Agile coach 1 (AC1) (10+ ys) Agile coach 2 (AC2) (5+ ys) Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 15 Case Industry, product offering & size Type of portfolio, number of agile teams & methods in use Agile practice history Case participants & agile experiences F Banking and financial services It provides various banking services, e.g., investment banking, credit and debit cards, finance, and insurance services. More than 33000 staff were working at the bank. Its projects ran under multiple portfolios. Technology risk portfolio The portfolio was structured under the technology and operation business unit and consisted of more than 50 agile teams (7-9 members each) using Scrum, XP, Kanban, and waterfall to deliver projects. The company has been adopting agile since 2010. Agile was first introduced in an e-Health project portfolio. From 2010 to 2013, the company transformed about 50 teams into agile under different digital platforms. Such adoption improved communication, teamwork, staff motivation, project transparency, and product time to market. The company has occasionally used various agile consultancy firms to train its agile teams. Two people: Portfolio manager (PfM) (5+ ys) Portfolio scrum master (PSM) (6+ ys) G Online real estate services It provides online real estate and commercial property advertising services. It had more than 1400 staff, and its projects ran under several portfolios. Technology portfolio The portfolio was structured under the market and technology development directorate and contained more than 30 agile (10 members each) teams using Scrum, XP, and Kanban for its projects. Sensing the pressures of the fast- paced and digitally connected world, the company started its agile journey in 2010. The company partnered with one leading agile consultancy firm to restructure its project delivery and core business systems. This partnership led to increased innovative delivery capability and collaborative, agile culture. The company’s entire business and ISD project portfolios practice agile. One person: Delivery lead (DL) (7+ ys) Table 3. Case descriptions The seven cases provided an appropriate mix for the study as they differed in their organisational structures, product offerings and services. Such an appropriate mix allowed us to gain insights from multiple perspectives, enhancing data triangulation (Etikan et al., 2016). As prior research suggests that project and portfolio managers’ skills, competencies and authority impact portfolio success (Beringer et al., 2013), we only interviewed experienced APM and PPM practitioners within the case organisations. 3.2 Data Collection This study’s primary data sources were face-to-face semi-structured interviews. Semi- structured interviews facilitated the collection of rich data while maintaining the flexibility required for an exploratory study (Saunders & Lewis, 2018). All interviews were conducted in the participants’ preferred settings. Interviews ranged from 45 to 75 minutes and were all audio-recorded. The audio recordings were later transcribed and sent back to the interviewees for cross-checking. Field notes were also taken during interviews, and clarifications were sought after the interviews when required. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 16 Before data collection, an interview protocol containing several introductory and core sections was developed (Yin, 2018). The introductory section contained questions about the positioning and structure of the ISD portfolio, the organisation’s agile journey and the participant’s roles and responsibilities regarding APM and PPM and, finally, methodologies used for APM under the portfolio. The core section focused on asking for in-depth information related to the research questions, and follow-up questions were asked based on probes when appropriate. The identified agile scale-up and project portfolio SFs from the previous literature (Tables 1 and 2) allowed us to set up appropriate interview probes. The questions asked during interviews are shown in Appendix 1. 3.3 Data Coding and Analysis The conceptual framework served as a priori construct to help shape the initial design of theory building process (Barratt et al., 2011; Eisenhardt, 1989). However, without having a theoretical conceptualisation that sufficiently explains agile portfolio success (Stettina & Hörz, 2015; Sweetman & Conboy, 2018), we decided to undertake an inductive approach to analyse data (Miles et al., 2014). In addition, previous studies on portfolio success do not sufficiently explain PPM’s application in different contexts (Martinsuo, 2013; Martinsuo & Geraldi, 2020), especially in agile project environments. Figure 2. The data analysis process used in the study Figure 2 depicts the data analysis procedure. In the analysis process, we organised interview data before processing. Organising data involved transcribing interviews, typing field notes and arranging data in NVivo 12 Software. The initial reading of the data concerned the authors reflecting on the overall relevance and credibility of the data, consulting with participants to clear doubts about transcriptions and adding descriptions/memos when needed. We referred to Miles et al. (2014) and Saldana (2016) to perform two coding cycles. We used initial coding for the first coding cycle to conduct within-case analyses. At this stage, we remained flexible Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 17 to all possible theoretical directions. However, we consistently referred to the research questions and conceptual framework to stay on track during the coding process. All first-cycle codes were tentative and provisional. We changed or reworded some codes as the analysis progressed. We also identified several themes from the codes and generated memos/descriptions that supported us in understanding the codes and themes as we transitioned to the second coding cycle. We employed pattern coding for the second cycle coding to categorise the codes and themes that emerged from the first coding cycle. The conceptually similar codes were combined throughout the process, while infrequent codes were reassessed for inclusion in the emerging framework. We conducted cross-case analyses to identify similarities and dissimilarities across cases and generated common themes and subthemes (Miles et al., 2014; Saldana, 2016). The common themes related to SFs were classified into two contingency categories based on CT: C-SFs and P-SFs (See Section 2.4). Then, we integrated these categories into the study’s agile portfolio success framework and moved on to present and discuss the research findings and their implications considering the theoretical lens and previous literature (Sections 4 and 5). An example of the coding process is provided in Appendix 2. 3.4 Process and Data Validation The study followed Miles et al. (2014) suggestions to ensure the quality of the research process, data and findings. We carefully designed the research process to strengthen reliability and trustworthiness and used multiple cases from various industries and product offerings. We ensured that interview questions and probes were aligned with the research questions, conceptual framework and literature review findings, allowing for a productive inquiry-based conversation. We tested the interview protocol by performing two preliminary interviews that further improved the cohesiveness and breadth of the interview questions and probes (Castillo-Montoya, 2016). We also ensured transparency and procedural rigour using computer-assisted coding software for data coding (NVivo 12). To gain insights from various perspectives and ensure data triangulation (Etikan et al., 2016), we chose cases from various industries and selected participants with differing roles and responsibilities in agile ISD portfolios. To ensure that the coding process was reliable, the first author performed the entire coding process while, at the same time, the second and third authors cross-checked the quotes, themes and the entire analysis process. In the next section, the research findings are presented. 4 Analysis of Findings Figure 3 depicts the agile portfolio success framework and integrates the research findings for RQ1 and RQ2. With the presented framework, we suggest that achieving success in portfolios of agile projects in large organisations depends on achieving agile portfolio SCs. Also, the identified SFs should be carefully considered and adequately addressed to improve the chances of achieving agile portfolio success. To address RQ1, four specific agile portfolio SCs (SC1 to SC4) were identified from the data analysis. These SCs help explain multiple facets of success in portfolios of agile projects in large organisations. We explain these SCs in detail and highlight the new findings in Section 4.1. To address RQ2, 15 agile portfolio SFs emerged from data analysis, of which nine are categorised as contextual SFs (C-SF1 to C-SF9) and the rest as project-level SFs (P-SF1 to P-SF6). In section 4.2, we present the SFs and highlight the new findings. Section 5 discusses the new findings and explains the research implications. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 18 4.1 Success Criteria for Agile Portfolios RQ1 is concerned with what constitutes agile portfolio success in large organisations. We asked research participants how their case organisations determined the success of their agile portfolios. Ten sub-themes emerged from the data analysis, which reveals how case organisations measure or determine the success of agile portfolios under study. These sub- themes were categorised into four main themes: portfolio value maximisation, strategic alignment, achieving economic success and portfolio balancing. Figure 3. The agile portfolio success framework Table 4 summarises the identified SCs, their subthemes and supporting references from the traditional PPM research. We specifically identified four new subthemes unique to agile portfolios. The following explains all the identified SCs and highlights new findings. Themes (SCs) Subthemes* Supporting references SC1. Portfolio value maximisation It denotes achieving maximum value for agile projects in a portfolio. Successful delivery of projects The criterion concerns achieving average project efficiency (satisfying budget, schedule, and specification) over all agile projects in a portfolio. (Kester et al., 2014; Kock et al., 2016; Martinsuo & Lehtonen, 2007; Meskendahl, 2010) Benefit for the customers The criterion measures the impacts on project (Martinsuo & Killen, 2014; Padovani & Carvalho, 2016; Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 19 Themes (SCs) Subthemes* Supporting references customers, e.g., satisfying customers and solving customers' problems. Voss, 2012; Voss & Kock, 2013) Increased customer involvement New It represents efforts to address increased customer interaction and focus on agile portfolios. - SC2. Strategic alignment It represents the degree to which agile projects reflect organisational strategy. Strategic alignment It refers to criteria set at the portfolio level to ensure the alignment of agile projects to the organisational strategies and goals. (Jonas, 2010; Kopmann et al., 2017; Meskendahl, 2010; Stettina & Hörz, 2015; Unger et al., 2012) SC3. Achieving business success It represents the contribution of an agile portfolio to overall business success through financial and non-financial achievements. Market performance The criterion is about how sales objectives such as sales volume and market share are achieved when launching products and services from agile projects in a portfolio. (Jonas, 2010; Kester et al., 2014; Killen et al., 2008b; Meskendahl, 2010) Shorter delivery to market New The criterion is related to tracking shorter releases of multiple agile project outcomes to the market/customer. (Suomalainen et al., 2015) – only partial support Embracing change New It measures the quantity of embraced (accomplished) changes resulting from projects delivered in an agile portfolio. - SC4. Portfolio balancing It is about a range of dimensions, e.g., resources, that must be balanced for an agile portfolio to provide the best value to the organisation. Resource allocation It is about balancing a portfolio through the effective utilisation of resources. (Jonas, 2010; Kaufmann et al., 2020; Kock et al., 2020; Meskendahl, 2010; Unger et al., 2012) Risk assessment It concerns the management and balancing of project risks in a portfolio. (Jonas, 2010; Kaufmann et al., 2020; Kock et al., 2020; Teller et al., 2012) Team capability building New The criterion measures the agile teams' productivity in an agile portfolio, ensuring that it increases over time. - Note. *Subthemes are not ranked in order. Table 4. Identified success criteria for agile portfolios SC1. Portfolio value maximisation Participants referred to achieving the maximum overall value of the portfolio as one SC for PPM through three different subthemes: - Successful delivery of projects can be achieved by ensuring agile projects’ efficiency. Project efficiency refers to the traditional criteria of success, i.e., budget, schedule and quality compliance (Serrador & Turner, 2015; Shenhar & Dvir, 2007). We found that agile project efficiency is tracked as a critical contributor to portfolio success in all cases. For example, one participant emphasised the agile portfolio success: “We still track the more traditional success measures. Did you do it on time? Did you do it on budget? Did you deliver on your scope? [sic]” [Case -A, PpM]. Our findings here are comparable with the traditional PPM research, which also considers average project success over all projects belonging to a portfolio as an SC (Kock et al., 2016; Meskendahl, 2010). Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 20 - Benefit for the customers was identified in all cases. This criterion is about assessing the value of a portfolio through the impacts agile projects have on their customers. Participants mentioned several customer-related criteria for agile project success, including satisfying customers, fulfilling customer needs, the product's actual use by the customer and solving a customer’s problem. For example, customer satisfaction was considered a portfolio success measure in Case A: “I would judge customer satisfaction as the primary driver of success” [Case-A, PpM]. Alternatively, the ADM in Case C believed that “If we can solve the sponsor’s problem and bring value to the customer, in that sense, we can say that we have been successful” [Case-C, ADM]. Previous research considers the impact on the customer as an SC for individual projects (Serrador & Turner, 2015). Based on our findings, we argue that an agile portfolio's benefit or impact on customers should be evaluated as an SC. The benefit for the customers can be determined by assessing the overall value delivered to customers in an agile portfolio. - Increased customer involvement New represents efforts to address increased customer interactions in agile portfolios. Customer involvement is perceived as crucial for agile portfolio success and was explicitly mentioned by participants in several cases (Cases-A, E and G) as an SC. For example, “A portfolio is there to increase the delivery pace and customer interaction as much as possible” [Case-E, AC1]. This SC is unique for the agile portfolios as previous studies have only discussed increased customer involvement for individual agile projects (Sheffield & Lemétayer, 2013) and have not considered it an agile portfolio SC. SC2. Strategic alignment Participants repeatedly referred to strategic alignment as one important SC through a single subtheme: - Strategic alignment denotes the degree to which the projects under a portfolio reflect organisational strategy. All participants mentioned their PPM efforts to align agile projects’ goals and objectives with organisational strategy. For example, “We have a strategy, and then the initiatives linked to the strategy. We have themes which are a level up close [sic] to the strategy and then, initiatives and features...” [Case-D, AC]. Strategic alignment was considered as adding value to the portfolio business and customers. Participants referred to several practices, such as establishing product visions, roadmapping and portfolio backlogs, that helped maintain strategic alignment at the portfolio level. Aligning projects with organisational strategy is a key factor in achieving portfolio success, as recognised by both traditional and agile project management literature (Kaufmann et al., 2020; Stettina & Hörz, 2015). SC3. Achieving business success Participants referred to achieving business success as one SC for PPM through three subthemes: - Market performance deals with the projects' outcomes and is about the extent to which sales objectives (e.g., sales volume or market share) are achieved by introducing the projects' products to the market. For example, a participant in Case B mentioned: “The real thing is: Have we solved a customer problem? Have we generated value for the customers? If yes, then over time, that enables us to expand our business” [Case-B, SM]. Market performance is about measuring the actual benefits of projects under a portfolio once they are introduced to the market (Killen et al., 2008b; Meskendahl, 2010). Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 21 - Shorter delivery to market New is an agile-specific criterion monitored in agile portfolios. Participants mentioned agile PPM attempts to monitor products’ time-to-market. For example, “In the traditional way, you cannot start measuring value until a long time [has passed]. If you are delivering value earlier, you can start gaining that value, actual income and results… One important thing is lead times. From the moment you conceive an idea until the moment you deliver it, the shorter is better” [Case-D, AC]. Such an SC is rarely discussed in traditional PPM literature. - Embracing change New represents the efforts of agile portfolios to measure if the changes are accomplished in a multi-project agile environment. Traditional PPM studies do not recognise such an SC at the portfolio level. Research participants in Cases B, F and G believed the number of accomplished changes in an agile portfolio should be considered a measure of success. For example, in Case F, the concept of “value drops” was introduced by PPM as an indicator of the number of embraced changes and was measured to ascertain if the agile portfolio can deliver more customer value over cyclical periods. “Based on our recommendation to the executive, we believe we could achieve 46 value drops. So, this year, we achieved 49. That is a [portfolio] success” [Case-F, PfM]. Traditional PPM studies do not recognise embracing change as a portfolio SC. SC4. Portfolio balancing This category contains three distinctive subthemes as an agile portfolio SC: - Resource allocation is about balancing each portfolio through the effective utilisation of resources. There was agreement among all participants concerning the importance of resource balancing in ensuring agile portfolio success. However, participants in Cases A, B and C referred to resourcing issues that hindered smooth portfolio delivery. For example, ineffective resource balancing was evident in Case B; “Sometimes we have too many software engineers and not enough product people, and other times, we have too many designers and not enough software engineers. So, there is definitely a levelling problem that we have from time to time…” [Case-B, VP]. We further discuss this criterion along with the risk assessment below. - Risk assessment addresses managing multiple agile projects’ risks and balancing their impact at the portfolio level. “The risk [management] is very important in ensuring that we deliver what we say. So we do risk assessment to ensure the risks are evaluated for risk rating and determining activation dates. For example, something [a software product] will be out of date today, but it will be horrible in 3 months. Because we will not have the patch, it [the service] will be out of support… We have a team to assess the risk, even before a project comes to our backlog” [Case-F, PSM]. For a portfolio to provide the best value to its organisation, it must be well-balanced across a few essential project-related dimensions, e.g., types, sizes, risk levels and resource demands (Abrantes & Figueiredo, 2015). A right portfolio balance allows organisations to achieve their objectives without being exposed to unnecessary risk and secure portfolio success (Kester et al., 2014; Meskendahl, 2010). However, determining what dimension to consider depends on the portfolio context (Meskendahl, 2010). We found two dimensions of portfolio balancing (i.e., risk and resources) prevalent in our studied cases. - Team capability building New was only measured in portfolios of several cases (Cases B, F and G). Evaluating a team’s capability was perceived as an attempt to increase the agile teams’ productivity over time, which, in the long run, helps organisations achieve higher portfolio success rates. “The whole point of having an enduring portfolio is, if we can do 49 [value drops] Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 22 this year, maybe we can do 65 next year. Because the teams will inevitably go faster and get better. So, did I improve the teams’ capabilities? If not, then I have not achieved my [portfolio] goal” [Case- F, PfM]. Building team capability contributes to sustaining agile portfolios by ensuring that the agile teams’ productivity increases over time. Such an SC is rarely discussed in PPM literature. 4.2 Success Factors of Agile Portfolios RQ2 asks about factors contributing to the success of agile ISD portfolios in large organisations. Data analysis revealed 15 themes (SFs) related to agile portfolios. As shown in Figure 3, nine were categorised as C-SFs, and the remaining six were P-SFs. C-SFs contain a range of organisational, cultural and process-related factors, while project-level SFs are project and team related. We also used the U and C dimensions of the UC Framework, explained in Section 2.4, to analyse the research findings. Table 5 shows the clustering of identified SFs based on the U and C contingency categories. SF category SFs related to u ncertainty (U) SFs related to consequences (C) Contextual SFs C-SF1. APM-PPM synchronisation C-SF2. Top management support C-SF6. Appropriate funding model C-SF3. Organisational culture C-SF8. IT and infrastructure agility C-SF4. Communicating agile values C-SF9. Flat organisational structure C-SF5. Team/customer involvement C-SF7. Customer training Project-related SFs P-SF1. Information availability and quality P-SF3. Team autonomy P-SF2. Clear project vision P-SF5. Team alignment P-SF4. Tailoring agile delivery metrics P-SF6. Sufficient amount of work Table 5. Categorisation of identified SFs based on the UC framework We found eight SFs related to uncertainty, complexity or urgency, which we categorise as U SFs. For example, IT and infrastructure agility (C-SF6) is a factor related to the complexity arising from the structure of a firm (or its PPM), so it falls under the U category. The remaining seven SFs were related to the consequences, such as the ability of teams to communicate, understand and quickly respond to unexpected events at the project or portfolio level (team empowerment); hence, they were categorised as C SFs. For example, team autonomy (P-SF3) designates the discretionary power given to agile teams and is a team empowerment contingency that falls under the C category. Table 6 summarises the identified SFs and supporting references from the agile scale-up and traditional PPM research. In the following, these SFs are presented based on their frequency of appearance in data analysis, with the most frequent appearing first. Subthemes Description Supporting references Theme: Contextual SFs C-SF1. APM-PPM synchronisation New Organisations need to align PPM with agile project delivery. This can be achieved at the portfolio level by implementing and supporting several agile activities, such as portfolio backlogs and stand-ups. For PPM (Sweetman & Conboy, 2018) C-SF2. Top management support Senior management engagement and support are found essential for sustaining agile portfolio success. For agile scale-up (Dikert et al., 2016; Edison et al., 2021; Kalenda et al., 2018; Rigby et al., 2018; Shameem et al., 2018; Stettina & Hörz, 2015); for traditional PPM Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 23 Subthemes Description Supporting references (Beringer et al., 2013; Blomquist & Müller, 2006; Jonas, 2010; Jonas et al., 2013; Kock et al., 2015; Rank et al., 2015; Unger et al., 2012) C-SF3. Organisational culture Important cultural characteristics such as trust, open communication and a no-blame environment facilitate a safe environment for agile teams to openly communicate issues, develop relationships and collaborate on agile projects. Such organisational culture contributes to the success of agile portfolios. For agile scale-up (Abrar et al., 2021; Edison et al., 2021; Kalenda et al., 2018); for traditional PPM (Kaufmann et al., 2020, 2021; Kock & Gemünden, 2020; Rank et al., 2015; Unger et al., 2014) C-SF4. Communicating agile values Constant communication of agile values influences organisational culture while enhancing communication among agile teams, projects and portfolios. For agile scale-up (Dikert et al., 2016; Kalenda et al., 2018; Rigby et al., 2018) C-SF5. Team/customer involvement New Agile teams and customers should play an active role in portfolio decision-making. Such active involvement enhances communication and collaboration, which eventually promotes portfolio success. For internal stakeholders of traditional PPM (Beringer et al., 2013; Blomquist & Müller, 2006; Jonas, 2010; Petro & Gardiner, 2015; Unger et al., 2012) C-SF6. Appropriate funding model New Milestone-based investment plans might hinder the successful delivery of agile portfolios. More flexible funding models, such as capacity-based funding, can allow better management of multiple agile projects. As partial support for agile scale- up (Edison et al., 2021; Hoda et al., 2009; Rigby et al., 2018) C-SF7. Customer training New Customers' lack of knowledge and awareness about agile methods limits their ability to collaborate effectively in agile projects. PPM should support and facilitate customer training to increase the chances of agile projects and portfolio success. As partial support for agile scale- up (Conboy & Carroll, 2019) C-SF8. IT and infrastructure agility New Legacy IT and infrastructure processes constrain successful agile delivery. PPM has a role in removing legacy issues to support the delivery of agile projects. As partial support for agile scale- up (Bosch & Bosch-Sijtsema, 2011) C-SF9. Flat organisational structure New Highly vertical organisational structures can hinder fast decision-making at the project and portfolio levels and undermine successful agile project delivery. A flatter (organic) PPM structure can better serve agile projects and portfolios. In the form of aligning the organisation in general for agile scale-up (Conboy & Carroll, 2019; Dikert et al., 2016; Edison et al., 2021); for agile PPM (Sweetman & Conboy, 2018) Theme: Project-level SFs P-SF1. Information availability and quality The availability and quality of information gathered from agile projects can contribute to portfolio success through enhanced communication and decision-making. For agile scale-up (Dingsøyr et al., 2019; Edison et al., 2021; Kalenda et al., 2018; Shameem et al., 2017); for traditional PPM: (Dietrich & Lehtonen, 2005; Jonas, 2010; Jonas et al., 2013; Lerch & Spieth, 2013; Martinsuo & Lehtonen, 2007; Spieth & Lerch, 2014; Teller et al., 2012) P-SF2. Clear project vision The clarity of project/product vision is an enabling factor for both agile project portfolio success. For agile scale-up: (Edison et al., 2021; Kalenda et al., 2018; Shameem et al., 2017); for traditional PPM: (Biedenbach & Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 24 Subthemes Description Supporting references Müller, 2012; Martinsuo, 2013; Martinsuo & Lehtonen, 2007) P-SF3. Team autonomy New There should be a proper balance between autonomy and control in PPM. For PPM to secure agile portfolio success, agile teams should be empowered to practice self- organisation and autonomy. For agile scale-up: (Dikert et al., 2016; Edison et al., 2021; Shameem et al., 2017); in the form of partial support for agile PPM (Sweetman & Conboy, 2018) P-SF4. Tailoring agile delivery metrics New Agile delivery metrics should be tailored for each individual project at the portfolio level. Then, the performance and fitness of each agile project should be consistently reported and checked against the pipeline of work. In the form of lack of agile metrics as a challenge for agile scale-up (Edison et al., 2021) P-SF5. Team alignment An effective team alignment guarantees an optimal balance between project requirements and resource availability, eventually promoting agile portfolio success. For agile scale-up (Edison et al., 2021; Kalenda et al., 2018; Rigby et al., 2018; Shameem et al., 2017; Šmite et al., 2017); in the form of stakeholder management importance for PPM (Dietrich & Lehtonen, 2005; Jonas, 2010; Kaufmann et al., 2020; Kock & Gemünden, 2020; Kopmann et al., 2017; Müller et al., 2008; Teller, 2013; Teller et al., 2014; Teller & Kock, 2013) P-SF6. Sufficient amount of work New Agile teams are often regarded as continuous product delivery teams. A continuous and sufficient pipeline of work supports agile teams in finding a consistent and harmonised rhythm of work that increases the likelihood of portfolio success. - Table 6. Identified success factors for agile portfolios 4.2.1 Contextual Success Factors C-SF1. APM-PPM synchronisation New – In the studied cases, adopting agile at scale resulted in adding several agile-specific activities (practices, artefacts and rituals) to the PPM. Participants mentioned various agile activities managed at the portfolio level, such as discovery workshops, iterative planning sessions, roadmapping, portfolio backlogs, portfolio walls, portfolio stand-ups and showcases. By performing these activities, organisations tried aligning PPM with agile project delivery. In most cases (Cases A, C, D, E and F), participants considered such efforts critical to achieving agile portfolio success. Participants also reported imbalances between APM and PPM that, in their view, frustrated agile teams and hindered agile project delivery. These imbalances were mainly related to highly formal and plan-based PPM governance structures. For example, in Case A, business cases and project funding models were milestone-based, forcing agile teams to report project changes and progress using traditional project measurement metrics. Such action was considered to be time-consuming and unnecessary. “I think we do struggle because our [PPM] framework is still a bit waterfall. We still go through initiatives and plans. The teams tend to be ever more agile, but it is just aligning against the [PPM] framework that is a barrier to success. From a PPM perspective, I think we are not quite aligned yet” [Case-A, PfM]. Failure to synchronise PPM with agile project delivery will compromise the ability of the agile Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 25 teams to react rapidly to changes in project requirements and business environment and adds to the complexity of managing agile projects at the project level. Hence, we considered it a factor related to urgency and complexity (U SF). C-SF2. Top management support – Across all cases, participants perceived management support as essential for sustaining portfolio success. For example, such support was evident in Case A: “Within our portfolio, they [top managers] are very much all on the same page for what we need to do to support the teams” [Case-A, RTE]. Continuous executive management commitment and support are crucial in sustaining organisational change (Pikkarainen et al., 2012) and setting up suitable and agile-friendly organisational roles (Stettina & Hörz, 2015). Previous research confirms management support as an SF for individual agile projects (Sheffield & Lemétayer, 2013), traditional portfolios (Unger et al., 2012) and agile portfolios (Leffingwell, 2007; Stettina & Hörz, 2015). This SF can be attributed to the organisational structure and culture, so it is a factor related to both complexity and team empowerment. However, we considered it a factor related to consequences (C SF) because of its profound influence on the agile teams’ ability to self-organise and deliver successful products (Sheffield & Lemétayer, 2013). C-SF3. Organisational culture – Data analysis revealed several cultural characteristics such as trust, open communication and a no-blame environment contributing to agile portfolio success. Participants expressed the importance of PPM in facilitating a healthy and safe environment for agile teams enabling them to openly express their issues, develop relationships, make mistakes and learn from those mistakes. For example, according to one participant, PPM provided such an environment: “I think everyone has to work in concert… I did mention about [sic] trust and giving freedom to teams by trusting them…” [Case-F, PfM]. The participants also stressed the importance of managers avoiding micromanagement to foster innovation. For APM to be adopted and sustained, an organisation's culture and agile values must be a good fit. An organisation's culture reflects the perceptions of managers and workers that form the prevailing mentality of the organisation. It reflects the values and beliefs of its staff (Schein, 2009). The organisational culture is a highly subjective matter that needs to be contextualised considering the purpose and strategy of the organisation (Tolfo et al., 2010). Traditional PPM research also recognises facets of culture, such as the innovation culture, as essential for portfolio success (Unger et al., 2014). Our findings highlight the importance of appropriate organisational culture in achieving agile portfolio success. Organisational culture ascertains the discretionary power formally assigned to the teams. Also it is an imposed factor that may hinder their ability to use this power effectively (Howell et al., 2010), so it is a C SFs. C-SF4. Communicating agile values – In all cases, intensive communication of agile values and principles at the portfolio level (and organisation-wide) was evident. Participants underscored the importance of communicating the agile way of working; for example, “Agile is more about the culture; capability can come with time… The hardest point was to change the culture. To say we are all owning something. If you see that it does not work, it is your problem as well. Capability as uplifting the agile knowledge comes if you are keen, willing to understand and accept the change. This is the message that needs to be communicated” [Case-F, PSM]. Communicating agile values influences organisational culture and enhances communication and collaboration across agile projects and their portfolio; thus, it is a team empowerment contingency and a C SF. Communication is a key enabler of agile projects Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 26 (Mishra et al., 2012). Constant communication of agile values is essential in raising awareness of agile at scale. It is more likely for people in an organisation to contribute effectively to promoting the agile way of working when they understand the agile values and mindset (Dikert et al., 2016). C-SF5. Team/customer involvement New – Participants stressed the active role and involvement of agile team members and customers in portfolio decision-making. For instance, in Case F, portfolio managers invited team members and customers to weekly portfolio meetings and involved them in making decisions about projects and portfolio issues: “At the portfolio level, we have another weekly meeting. All key project managers and customers are invited. We use these weekly stand-ups to make quick decisions where we can. So, those meetings are not just progress updates. The key difference is that at those meetings, we actually make decisions” [Case- F, PfM]. Team/customer involvement in portfolio decision-making can be attributed to the level of complexity at the portfolio level. However, we argue that its dominant effect is in modifying the agile teams’ and portfolio managers' ability to process and act effectively upon information received, so it can be considered a C SF. C-SF6. Appropriate funding model New – In most cases (all except Cases B and G), agile portfolios ran under centralised governance structures characterised by annual budgeting cycles and milestone-based business/investment plans. Participants identified mismatches between agile delivery outcomes and business/investment plans when project changes surfaced during its execution. Research participants proposed capacity-based funding as an appropriate funding model promoting agile projects and portfolio delivery in large organisations. “One way is the capacity funding model, where you fund all the persistent teams… You do it away from the business cases and put more autonomy into the product and business owners to have a fixed funding model, which is capacity-based. Then, you prioritise the work into those teams” [Case-C, AC]. In our view, the funding model for projects is a contingency influencing project goals, which is a factor related to uncertainty. Therefore, we considered it as a U SF. C-SF7. Customer training New – A shared view among the participants was that customers need agile training as much as agile teams do. According to the participants, the lack of customer awareness about agile methods impeded active customer collaboration and negatively impacted agile projects and portfolio success. Participants in Cases A, E and F revealed efforts from agile teams, supported by portfolio managers, to involve customers in agile ceremonies and rituals to train them on the agile way of working. “Some [customers] still do not know [agile] very much. So, part of our job is ensuring that all stakeholders we start working with - we give them the context of how we work and what is actually done so that they feel comfortable working in that way with us” [Case-A, RTE]. Training customers in agile improves their ability to delineate project goals and requirements (an uncertainty factor) and remove scepticism about the agile way of working (a team empowerment factor) that positively impacts team-customer collaboration (Hoda et al., 2011). We considered this factor a C SF. C-SF8. IT and infrastructure agility New – Participants in Cases C, F and G referred to legacy IT and infrastructure processes that constrained successful agile delivery. APM generally requires dealing with organisational and IT legacy processes and software. Sometimes, customer value(s), such as building new product features, must be built on a legacy process. It is also plausible to replace a legacy process with a new one to support agile methods (Thomas & Baker, 2008). Participants believed that one portfolio manager’s job Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 27 was to remove legacy issues to support the delivery of agile projects. However, they also expressed concerns that upgrading or replacing these processes could become challenging, time-consuming and costly. A participant explains a legacy system problem: “Setting up the IT systems for agility is another biggest [sic] limiting factor. The state of the software in the organisation is a big limiting factor in how agile you can be. I had many organisations over the years that put different [agile] products and systems [sic]. They are not set for agility, so if the business wants to build a new feature, it can be developed digitally. However, it has a dependency on a traditional system that has a very long lead time. That impedes success” [Case-C, AC]. We consider this factor a U SF since it is a contingency related to organisational structure that can influence the level of complexity at the portfolio level. As far as we know, IT and infrastructure agility is not considered an agile portfolio SF in previous literature. C-SF9. Flat organisational structure New – In our study, only two cases (Cases B and G) benefited from flat organisational structures - also called organic structures (Donaldson, 2001). Other cases had highly vertical structures. Participants in vertical structures repeatedly expressed concerns about structural mismatches that undermine successful agile delivery. These mismatches mainly surfaced when changes in agile projects went through lengthy decision-making processes and were compared with traditional milestone-based business cases for approval. Cases with flat organisational structures, on the other hand, were able to facilitate rapid communication channels, easy access to sponsors and speedy decision- making at project and portfolio levels. Participants had a common belief that a flatter organisational structure better serves agile portfolios. A participant in Case-C explains the structure: “In this organisation, there is no sense of hierarchy. There is, but it is very minimal. Where things go wrong is when people on the frontline take instruction from the senior people without challenging it. When that happens, it is a learning moment for everybody involved because the senior person gave the instructions [but] failed to deliver a line that says, “I am not an expert; this is just an opinion; you need to make a decision”. Plus, the people on the team took that senior person’s opinion as an instruction. If that happens, we consistently have problems. So as a result, we vigorously fight against the behaviour and remind people on the frontline that they are the decision-makers and are all accountable and responsible for the outcome” [Case-B, VP]. The cases with flat organisational structures in their agile portfolios reported fewer issues in achieving portfolio success. A flat organisational structure is characterised as a management structure with low functional formalisation where employees exercise autonomy in decision-making and self-organisation in performing their duties (Donaldson, 2001). This factor is directly related to organisational complexity and indirectly to team empowerment, so is a U SF. 4.2.2 Project-level Success Factors P-SF1. Information availability and quality – All participants believed project information availability and quality are essential for agile portfolio success. Participants discussed issues concerning information gathering and project data quality that negatively impacted the management of portfolios. For example, in Case F, delays in receiving timely updates, as well as low-quality and outdated information, hindered agile portfolio success by impeding effective decision-making: “Organising data sources, mapping all the information and making it like a core is very hard… There are many projects, tools, and information. What are your sources of truth, and what is the cadence of updating those? If you want to have a snapshot of the portfolio, you need to make sure the provided data are accurate. Some delivery teams are not Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 28 ready yet…” [Case-F, PSM]. Information quality enhances decision-making quality and portfolio success (Dietrich & Lehtonen, 2005). Therefore, PPM processes must be carefully designed to enable the sharing and communication of high-quality information. Information availability and quality can be associated with the method of project delivery, multiple projects’ goals and PPM design that supports effective communication. Hence, it is a contingency related to uncertainty and a U SF. P-SF2. Clear project vision – All research participants referred to the positive influence of a clear project/product vision in enabling agile projects and portfolio success. Participants pointed out several activities relevant to project visioning, such as discovery workshops and roadmapping. Project visioning was perceived as a way to align agile projects with portfolio goals and organisational strategy. A participant underscores the importance of project visioning: “I think it [portfolio success] starts with a clear vision and strategy at the leadership level. Once we have it, that needs to be translated into portfolios or larger project programs. So, you would have a clear vision and strategy, then a laundry list of projects and processes that underpin that vision” [Case-C, AC]. Without a clear project vision, the chances of problematic product integration at final testing increase, resulting in extremely costly and time-consuming rework in the end (Vlietland & van Vliet, 2015). Project goal setting is considered a critical SF in traditional PPM (Martinsuo & Lehtonen, 2007) and agile projects (Serrador & Pinto, 2015; Vlietland & van Vliet, 2015). Our findings reveal clear project vision as an agile portfolio SF related to uncertainty, which is a U SF. P-SF3. Team autonomy New – Team autonomy is perceived as “the degree of discretion and independence granted to the team in scheduling the work, determining the procedures and methods to be used, selecting and deploying resources, hiring and firing team members, assigning tasks to team members and carrying out assigned tasks” (Lee & Xia, 2010, p. 90). All participants emphasised the importance of embracing and facilitating agile teams’ self-organisation and autonomy at the portfolio level. In our participants’ view, PPM must empower agile teams to plan and organise their own work and assist them in making conscious decisions about their projects. Supporting autonomy was prevalent in Case-A: “They [portfolio managers] honestly just leave us alone. They do not interfere too much with what we do. It is very easy for them to have a lot of red tape around what we do and really make our lives hard, but they do not” [Case-A, RTE]. The team’s flexibility in making project decisions and managing their workload signifies procedural empowerment, which is a C SF. To our knowledge, supporting team autonomy as an SF for agile portfolios is not mentioned in PPM research. P-SF4. Tailoring agile delivery metrics New – In all studied cases, agile teams communicated their progress to PPM based on agile project metrics, such as burnup/burndown charts, velocities, lead times and cycle times – see Kupiainen et al. (2015) for various agile project metrics. For example, the use of team velocity and burn up/down charts were common across all agile teams in all cases. Data analysis also revealed a tendency to tailor agile metrics for individual projects under portfolios, especially during the early stages of the project conception. “At an early stage, once we have done the initial discovery or screening phase, we set up some criteria, and it is going to be slightly different for each project… We rather measure success on tangible outcomes” [Case-F, PfM]. Research participants believed that the performance and fitness of each agile project should be consistently reported and checked against the pipeline of work at the portfolio level. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 29 Another finding is related to using traditional portfolio metrics. The use of traditional project planning and risk control metrics was still prevalent in most cases (all except Cases B and G). In most cases, portfolio managers were still preparing plan-driven project status reports based on data collected from agile teams. In other words, portfolio managers translated the agile status reports into milestone-based reports for further use by senior management. The coexistence of traditional and agile metrics was non-value-adding, time- consuming and costly and was considered a waste of resources by several research participants. “I think [portfolio] metrics need to evolve to take into account your working and delivering agile stuff [sic]. We need to move away from some of the more traditional metrics we used to use… They [senior management] need to be familiar with the artefacts we are using. You may not need a full-blown project management plan anymore” [Case-A, RTE]. While this research does not provide specific recommendations for using specific type of metrics at the project or portfolio level, we highlight the importance of adopting appropriate agile delivery metrics in the context of agile portfolio management. Tailoring agile delivery metrics helps establish appropriate project and portfolio goals and objectives; therefore it is a factor of uncertainty and we considered it as a U SF. At the project level, appropriate metrics provide a reliable and accurate means of measuring the success of agile practices and identify areas that require improvement. The lack of appropriate metrics at the portfolio level can lead to poor decision making and project outcomes. Hence, it is imperative that organisations recognise the need to implement relevant metrics at project and portfolio levels to ensure the alignment of agile projects with organisational goals. P-SF5. Team alignment – Multiple agile projects generate substantial changes, producing sequences of cascading effects on their portfolio, thus demanding ever-changing decisions. The momentum created by these changes leads to resource conflicts, as all projects compete for limited resources (primarily human resources). As signified by the participants, a continuous and effective team alignment guarantees an optimal balance between project requirements and resource availability that eventually promotes agile portfolio success. Effective resource (team) allocation and alignment at the portfolio level were considered vital in ensuring agile team performance, finding a steady rhythm of work and achieving portfolio goals: “It [resource planning] is very systemic. In that, we staff teams on an annual budget. We make minor corrections during the year, but otherwise, that is [a] steady state. We do not change team sizes. There are only strategic moments where we will spin up a new team” [Case- B, VP]. Patanakul and Milosevic (2009) suggest that managing resources for individual projects should not be done in isolation. Instead, PPM requires a systematic and proactive approach to managing resources. Effective resource management is found as a critical SF for traditional PPM (Abrantes & Figueiredo, 2015). An effective team alignment can support agile teams in achieving cross-functionality that boosts team performance (Hoda & Murugesan, 2016) and influences the success of agile projects and portfolios. Team alignment is a contingency related to the team composition and capability (Howell et al., 2010), so we considered it a C SF. P-SF6. Sufficient amount of work New – Agile teams were considered continuous product delivery teams in all studied cases. Cases A, C and E participants mentioned portfolio managers’ efforts to feed portfolio pipelines with enough projects. There was a common belief among research participants that a continuous and sufficient pipeline of work supports agile Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 30 teams in finding a consistent and harmonised rhythm of work that increases the likelihood of portfolio success: “It costs money to keep each agile team running, so I have to have a backlog of work ready to go. If I do not have that backlog or approval somewhere or some business cases are sitting on my desk, I lose [teams’] efficiency” [Case-A, PfM]. This factor can be attributed to external factors (e.g., availability of projects in the business market) or internal factors (such as the need and extent of change in the organisation), both conditions related to uncertainty. Therefore, we considered this factor as a U SF. Such SF is rarely discussed in previous research. 5 Discussion of Findings 5.1 What Constitutes Agile Portfolio Success? The RQ1 asked about the criteria of success in portfolios of agile ISD projects in large organisations. To clarify the notion of success, we identified four SCs for portfolios of agile ISD projects in large organisations: portfolio value maximisation, strategic alignment, achieving business success and portfolio balancing. These SCs comprise ten subcriteria; among them, four are new and exclusive to agile ISD portfolios (Table 4). These new success criteria are increased customer involvement, shorter delivery to market, embracing changes and team capability building. In the following, we discuss the new SCs of agile ISD portfolios in light of extant literature. - Increase customer involvement – Quickly responding to customer needs and ensuring customer involvement are paramount to agile project success (Misra et al., 2009; Sheffield & Lemétayer, 2013). Agile projects strive for high customer satisfaction (Serrador & Pinto, 2015). Agile teams’ increased focus on customers brings complexities to the portfolio level. A task of portfolio managers would be flexibly adapting to and managing the tensions between customers’ needs and organisational strategies (Sweetman & Conboy, 2018). Research participants considered increasing customer involvement as one PPM objective and a measure of portfolio success. Increasing customer involvement at the portfolio level improves customer interactions with agile teams and ultimately benefits the agile projects and, therefore, the portfolio. - Shorter delivery to market – One crucial goal of an agile project is to reduce the product’s time to market (Cockburn & Highsmith, 2001). Rapid and incremental product launch to the market enables a faster reaction to market changes and customer demands, allowing organisations to create revenue faster and giving them a competitive edge over competitors in the long run (Cockburn & Highsmith, 2001). The shorter development time for projects is also considered essential to secure a consistent pipeline of work for project portfolios (Biedenbach & Müller, 2012). Portfolios dealing with agile projects increasingly need shorter delivery cycles to adapt to rapid changes in the business environment and customer demands (Suomalainen et al., 2015). - Embracing change – Such a criterion is considered an inherent goal of any agile project emphasised by the agile manifesto of software development (Beck et al., 2001). Therefore, for portfolios of agile projects, it is logical to consider and measure metrics related to embracing changes. Among the studied cases, we observed a tendency to quantify the number of changes embraced (accomplished) by multiple agile projects. Then, the number of changes embraced was used as a metric/benchmark to track the portfolio performance during portfolio review cycles. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 31 - Team capability building – We found portfolio managers' concerns and attempts to measure and improve agile teams’ capability over time as an SC for agile portfolios. For example, management in Case F introduced an indicator ‘value drop’ to measure the number of embraced changes which was also perceived as an indication of team capability building. Recent research points to the importance of team capability building for agile projects and portfolio success. For example, Kaufmann et al. (2020) studied the positive link between agile capabilities and emerging strategy recognition, and the latter is positively and significantly related to portfolio performance (Kopmann et al., 2017). Agile team capabilities are determined by the intensity of APM use and the team’s competence in applying agile practices (Kaufmann et al., 2020). Also, a recent study by Bechtel et al. (2021) demonstrates the positive influence of PPM practices and innovation culture on agile teamwork quality (enhanced communication, cooperation, support and commitment). However, such an agile portfolio SC is rarely discussed in previous literature. 5.2 What Factors Contribute to Agile Portfolio Success? The RQ2 asked about the factors contributing to the success of agile ISD portfolios in large organisations. We identified 15 SFs for agile ISD portfolios and categorised them into nine contextual SFs (C-SFs) and six project-level SFs (P-SFs), see Figure 3. The C-SFs are contingency factors related to agile portfolio environment, which are associated with the organisational structure and culture affecting portfolio success. We considered the P-SFs as contingencies related to individual agile projects and teams as portfolio subunits influencing agile portfolio success. Hence, the study proposes several new SFs contextual to the ISD portfolio as well as at the agile project level in large organisations. APM-PPM synchronisation, team/customer involvement, appropriate funding model, customer training, IT and infrastructure agility, and flat organisational structure are identified as new C-SFs. Also, team autonomy, tailoring agile delivery metrics and sufficient amount of work are suggested as new P-SFs. In the following, we discuss the new findings in light of previous literature and explore the effect of U and C SFs on determining appropriate portfolio structure. 5.2.1 New Contextual Success Factors C-SF1. APM-PPM synchronisation – This factor was identified as one key SF related to agile portfolio success. The PPM function in an organisation is to ensure projects have successfully delivered value and their outcomes align with business strategy (Martinsuo & Killen, 2014). The success of projects as portfolio components is crucial for portfolio success. At the same time, the ability to change is fundamental to the concept of agility at the project level. Agile projects constantly evolve in response to the projects’ changing requirements and environments (Schwaber, 2004). Therefore, PPM must fit the dynamic environment under which multiple agile projects constantly evolve (Sweetman & Conboy, 2018). Dynamic environments require fast responses to change that can be achieved through flexible organisational structures (Donaldson, 2001). Therefore, a flexible and adaptable PPM structure is expected to enable agile project delivery. It is also essential for PPM to adapt to several APM characteristics, such as high levels of team/customer interaction and team self-organisation and autonomy (Sweetman & Conboy, 2018). Portfolio managers, in the studied cases, made efforts to adapt PPM to APM by integrating (or supporting) various agile scale-up activities, including requirement discovery workshops, iterative planning sessions, roadmapping, portfolio backlogs, portfolio walls, portfolio stand-ups and showcases. Also, research participants showed awareness of the Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 32 need to measure and aggregate multiproject performances at the portfolio level. While the importance of agile scale-up activities is discussed in previous literature (Kalenda et al., 2018; Olszewska et al., 2016), APM-PPM synchronisation is rarely discussed as a critical SF for agile project portfolios. C-SF5. Team/customer involvement – ISD projects generally require the involvement of many stakeholders across multiple organisational disciplines (Dingsøyr et al., 2019). When multiple agile ISD projects are managed under a portfolio, the intensified team and customer interactions are aggregated at the portfolio level. Such aggregated interactions bring tensions and complexities to the portfolio level that needs to be coordinated and managed by PPM to ensure portfolio success (Sweetman & Conboy, 2018). Team/ customer involvement in PPM brings visibility to agile projects and portfolio decision-making. Previous literature suggests “engaging everyone” as one SF for large-scale agile scale-up efforts (Dikert et al., 2016, p. 102). Traditional PPM research also stresses the importance of internal stakeholders' involvement in PPM decision-making (Beringer et al., 2013). We identified that involving external stakeholders (such as customers) in PPM processes is crucial and innovative for agile portfolio success. C-SF6. Appropriate funding model – Our study suggests that a capacity-based funding model is more likely to promote agile portfolio success. Capacity-based funding distributes available funding across persistent, autonomous teams. Hence, funding is allocated based on the teams’ delivery capacity and the resources required to deliver the project results’ results rather than projects. Then, the agile teams prioritise work within the iterations and estimate the hours required to commit to set activities. Highly plan-driven business cases and fixed-term contracts are found to be incompatible with agile project settings (Cao et al., 2013; Hoda et al., 2009). These forms of funding are likely to impede the adaptability required to manage multiple projects at the portfolio level. More flexible types of contracts, such as negotiable scope and pay-as-you-go, are proposed for agile projects (Hoda et al., 2009). Capacity-based funding allocates funds across agile teams based on their delivery capacity and resource needs rather than the projects’ business cases and initial plans. C-SF7. Customer training – Effective stakeholder involvement is essential for the success of agile projects (Bass & Haxby, 2019; Kasauli et al., 2021; Misra et al., 2009) and is for agile portfolios. However, it is not uncommon for stakeholders to lack the essential knowledge of agile methods and mindset or are against them. Customer involvement at the project level can become more effective when stakeholders (especially customers) who are sceptical about agile's effectiveness or lack agile knowledge are trained. The need for customer training is intensified when dealing with multiple agile projects in portfolios of large organisations with many stakeholders. The research participants emphasised the significant role of PPM in identifying and assessing the need for customer training and facilitating it. While much has been said about the value of training and coaching for agile teams (Conboy et al., 2011; Gandomani et al., 2015), customer training is rarely addressed in agile and traditional PPM studies. C-SF8. IT and infrastructure agility – We found evidence of legacy IT and infrastructure processes that constrained successful agile delivery within the studied cases (e.g., Case C). The legacy processes often resonate with the legacy mindset, which is at odds with the agile way of working (Kalenda et al., 2018). Examples of likely legacy mindsets are upfront planning and order-taking. Upfront planning is the desire to think through everything Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 33 before starting the project, and order-taking is about waiting for orders and directions to perform a task (Thomas & Baker, 2008). Failure to address legacy issues and their underlying mindset generally leads to heavy top-down processes, ineffective resources and a lack of predictability and engagement (Bosch & Bosch-Sijtsema, 2011). Hence, assessing legacy IT and infrastructure are required before adopting agile at scale and in PPM. Failure to address legacy issues might lead to the cancellation or abandonment of agile scale-up efforts. C-SF9. Flat organisational structure – Large organisations require careful consideration of this contextual SF when dealing with agile portfolios. The cases with organic structures in their agile portfolios (such as Cases B and G) reported fewer issues in achieving success. These issues were related to inconsistencies between organisational (and portfolio structures) with agile project delivery that impedes effective coordination and communication. When scaling agile in vertical structures, agile teams' flexibility, autonomy and self-organisation are at odds with well-established traditional processes and routines of the organisation, causing issues for successful agile delivery (Sweetman & Conboy, 2018). On the other hand, flat organisational structures, i.e., organic structures, are characterised by flexible operating styles, open flow of information, less management control, more individual autonomy and better adaptation to changing circumstances (Donaldson, 2001; Slevin & Covin, 1990). An organic structure enhances coordination and communication and provides better access to organisational resources, stakeholders and leadership (Donaldson, 2001). Flexible self-organising structures can also respond quickly to the dynamic environments in which agile projects and portfolios operate. Based on our findings, we suggest that a flat (organic) portfolio structure can better adapt to unstable environments and requirements uncertainties associated with multiple agile projects, increasing the chances of agile portfolio success. 5.2.2 New Project-level Success Factors P-SF3. Team autonomy – Based on our findings, we proposition facilitating and supporting the project-level team autonomy at the portfolio level. Such facilitation allows development teams to manage their workload and practices, creates a sense of commitment, improves inter-team collaboration and empowers them to take ownership of the project delivery process. Conversely, portfolio managers' excessive control and micromanagement might demotivate teams and reduce productivity. Sweetman and Conboy (2018) argue that top- down control in an agile portfolio should be limited, and it is essential that “the portfolio purpose is created and shaped by both portfolio managers and individual agile projects” (2018, p. 28). PPM should provide a favourable environment for self-organisation to emerge while making teams collectively accountable for agile projects’ outcomes. The importance of team autonomy for effective delivery of agile projects (Hoda et al., 2011, 2013; Hoda & Murugesan, 2016) and for achieving software development agility (Sheffield & Lemétayer, 2013) has been emphasised before. Team autonomy creates commitment to agile scale-up efforts and motivates teams to take ownership of their project delivery efforts (Dikert et al., 2016). Balancing autonomy with control is also reported as an enabler of several agile scale-up frameworks, e.g., the Spotify model (Edison et al., 2021). P-SF4. Tailoring agile delivery metrics – Our research participants recommended careful consideration of agile-specific metrics for each individual project under an agile portfolio. Such an action allows for smooth monitoring and control of agile projects’ performances, Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 34 ensuring strategic alignment is achieved. According to the research participants, these metrics may include burn-down charts, velocities, lead times and cycle times, as well as metrics related to the product (software) performance and team productivity. In most cases, portfolio managers attempted to aggregate agile project metrics into plan-driven and milestone-based project status reports for further use by upper management. Most research participants perceived such an attempt as non-value-adding, time-consuming and costly. The research participants also recognised the importance of tracking project performance and alignment across various agile-specific portfolio metrics. More research is warranted to determine what portfolio metrics to use for agile portfolios. Tailoring agile delivery metrics is rarely discussed in agile project and portfolio literature. While traditional PPM research agrees on the importance of setting up appropriate project-level metrics, there is little agreement on what specific metric to choose, primarily when PPM deals with innovation projects (Killen et al., 2008b). Also, determining meaningful project metrics for performance review and improvement of agile projects is identified as a vital challenge for agile scale-up frameworks (Edison et al., 2021). Most of these frameworks have not provided clear recommendations for selecting appropriate project delivery and portfolio metrics (Edison et al., 2021). P-SF6. Sufficient amount of work – To our knowledge, such a necessary SF is rarely discussed in previous research. Previous agile literature refers to several workload challenges, such as overloaded teams and teams committing to unreasonable work (Dikert et al., 2016). Too much workload results in agile teams losing focus and productivity (Lee & Xia, 2010). On the other hand, in our participants' views, not having enough projects in the portfolio pipeline impedes continuous agile delivery, destroys the consistent rhythm of work, decreases team cohesiveness and alignment, and eventually forces the organisation to disband agile teams. 5.2.3 The Effect of Uncertainty and Consequences SFs As explained in Section 4.2 and summarised in Table 5, the identified SFs were either related to uncertainty, complexity and urgency, categorised as U SFs, or were linked to team empowerment and criticality, which we called them C SFs. Also, in Section 2.4, we argued that the UC Framework could be used to identify which process models (i.e., agile or plan-driven) are suitable for managing a particular project or portfolios of multiple projects. This section describes how classifying SFs into U and C categories can help structure an organisation's PPM. According to Howell et al. (2010), the U and C dimensions are orthogonal. Therefore, it is possible to select a process model based on whether U dominates C or vice versa. The dominance of U factors over C factors means that the degree of uncertainty, complexity and urgency in a multiproject environment is high, so an agile portfolio structure can be selected. For example, in such a case, the organisation can opt to select a flat organisational structure (C-SF9), work toward changing to an agile-friendly organisational culture (C-SF3) by constantly communicating agile values (C-SF4) and involving team and customers in portfolio-decision making (C-SF5). Also, the PPM should authorise iterative and incremental project delivery (APM) by encouraging project-level autonomy (P-SF3) and carefully tailoring project delivery metrics (P-SF4). Conversely, when USFs are not dominant or non-existing, a plan-driven and milestone-based portfolio structure can be applied. Therefore, portfolio and Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 35 project managers should focus on the upfront planning of multiple projects (i.e., employing waterfall methods) and seek optimal sequencing of activities and resources. 5.3 Research Implications 5.3.1 Theoretical contribution The identified agile portfolio SCs, SFs and agile portfolio success framework contribute to the theoretical conceptualisation, applicability and clarity of adopting agile methods at scale. The identified agile portfolio SFs can further enhance our knowledge and understanding of agile ISD portfolios in large organisations. Previous research narrates success stories of agile portfolios but provides little empirical evaluation of such portfolios in practice (Stettina & Hörz, 2015; Sweetman & Conboy, 2018). While the adoption of APM in portfolios of large organisations is increasing, empirical investigation on the usefulness and efficacy of these methods at scale is still very scarce (Conboy & Carroll, 2019). Hence, by introducing contextual and project-level SFs (C-SFs and P-SFs), this study contributes to understanding the portfolios of agile ISD projects and the conditions under which these portfolios achieve success. We have identified several SCs that determined agile portfolio success in case organisations. We argue that if success is sought for an agile portfolio, it is essential to carefully address the identified SC by setting up appropriate metrics, as well as considering the set of identified C-SFs and P- SFs when scaling up to agile portfolios. The research findings emphasise the importance of context as well as portfolio stakeholders in designing and managing agile portfolios. Hence, this study suggests that agile portfolio processes and practices should be carefully tailored to fit the organisation's working conditions and environment, considering the agile portfolio SCs and SFs. For example, PPM processes can be tailored to assist project teams in achieving clear project goals by implementing and supporting project discovery workshops and involving portfolio stakeholders (e.g., customers and teams) in PPM decision-making. Determining what agile projects and portfolio success mean to stakeholders is critical in defining explicit agile portfolio goals and SCs. 5.3.2 Managerial Implications The research findings have several managerial implications. The proposed agile portfolio success framework and the U and C categorisation of SFs can guide large organisations when adapting and sustaining agile methods at scale and in portfolios of ISD projects. For example, an agile portfolio structure can be appropriate if the organisation’s assessment determines the dominance or existence of multiple U SFs for a multiproject environment. In such a case, an organisation can think of ways to synchronise PPM with the management of agile projects and make portfolio managers responsible for advocating and communicating agile values and mindsets across projects and the rest of the organisation. Besides, for the success of an agile portfolio, it is essential to tailor delivery metrics for each individual agile project and find appropriate ways to aggregate these metrics for portfolio reporting. Also, broader team empowerment initiatives such as team autonomy and alignment are important and should be considered and supported by PPM. Checklists or metrics could be designed based on the proposed SFs to ensure the alignment of different organisational components (e.g., strategy and structure) with the identified SFs. Measuring performance based on these checklists or metrics could help organisations assess the effectiveness of performed actions and practices Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 36 during and after agile adoption at scale and the portfolio level. Organisations can then adjust their team and project agility efforts to support portfolio management. 5.4 Limitations While this research opens a new direction in terms of the adaptiveness of PPM to agile project environments, it also has limitations. The research participants were all experienced and knowledgeable in their fields; however, their standpoints might not fully represent their organisations. Although the cases were carefully selected, and we employed a rigorous method to ensure the research process and data validity, the findings are only based on cases and a limited number of participants from agile ISD portfolios in Australia, which may limit the general applicability of the findings. SAFe was mentioned in three cases (Cases A, D and E) as the agile scale-up framework, while the rest did not refer to any specific scale-up framework. Although the framework developed in this study is generic, it may be less relevant to organisations and practitioners using other agile scale-up frameworks. When developing the study’s framework, we assumed that all identified SFs are linked to agile portfolio success. However, the study was limited in showing the significance of the relationship between SFs and the identified SCs. Also, we did not investigate the dependence and power relationships between the identified SFs, which can be explored by future research. Finally, our study was limited in identifying appropriate agile portfolio metrics based on the identified agile portfolio SCs. Future research can examine appropriate portfolio level metrics for PPM. 6 Conclusion The popularity of agile methods in large organisations has increased over the past decade. However, a thorough understanding of adopting agile at scale and its impact on agile portfolio success is yet to be sought. This study empirically explores the notion of success in portfolios of agile ISD projects to determine its SCs and related SFs. Based on our findings, we propose that when agile ISD projects are managed under portfolio(s), traditional SFs are insufficient to secure portfolio success. Investigating seven large organisations, we identified four agile portfolio SCs (with ten subcriteria) and 15 SFs, integrating them into an agile portfolio success framework (Figure 3). The framework extends our understanding of agile portfolio success in large organisations. We argue that to achieve success in portfolios of agile ISD projects, large organisations should optimally fit several organisational characteristics (e.g., structure and strategy) with multiple contingencies (SFs) identified in this study. The agile portfolio success framework can guide large organisations through adapting and sustaining agile methods at scale and in their portfolios of agile ISD projects. Future work is needed to confirm the efficacy of the study’s framework using additional empirical tests. In particular, future research can utilise a broader sample size to corroborate the findings or can extend the findings to other contexts (e.g., agile projects in non-ISD environments or organisations with other types of agile scale-up frameworks). In addition, although our study explores SFs as contingency factors, a company’s PPM design is not exogenous and depends on external environmental conditions. Previous studies on PPM have investigated the importance of agility to portfolio decision-making and success in turbulent external environments (Kester et al., 2014; Kock & Gemünden, 2020), but future research could consider the fit between our identified SFs and portfolio-level agility. Finally, investigating the interdependencies between the identified SFs and determining the dependence and power relationships between SFs could potentially complement this research. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 37 References Abrantes, R., & Figueiredo, J. (2015). Resource management process framework for dynamic NPD portfolios. International Journal of Project Management, 33(6), 1274–1288. https://doi.org/http://dx.doi.org/10.1016/j.ijproman.2015.03.012 Abrar, M. F., Ali, S., Majeed, M. F., Khan, S., Khan, M., Ullah, H., Khan, M. A., Baseer, S., & Asshad, M. (2021). A framework for modeling structural association among de- motivators of scaling agile. Journal of Software: Evolution and Process, 33(8), 1–19. https://doi.org/10.1002/smr.2366 Agile-Business. (2017). Agile portfolio management ( AgilePfM ) - For the agile-mature organisation. Agile Business Consortium. https://www.agilebusiness.org/ Barratt, M., Choi, T. Y., & Li, M. (2011). Qualitative case studies in operations management: Trends, research outcomes, and future research implications. Journal of Operations Management, 29(4), 329–342. https://doi.org/10.1016/j.jom.2010.06.002 Bass, J. M., & Haxby, A. (2019). Tailoring product ownership in large-scale agile projects. IEEE Software, 58–62. DOI: https://doi.org/10.1109/MS.2018.2885524Cha Bechtel, J., Kaufmann, C., & Kock, A. (2021). Agile projects in nonagile portfolios: How project portfolio contingencies constrain agile projects’ teamwork quality. IEEE Transactions on Engineering Management, 1–15. https://doi.org/10.1109/TEM.2021.3130068 Beck, K., Beedle, M., Bennekum, A. van, Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for agile software development. http://www.agilemanifesto.org/ Beecham, S., Clear, T., Lal, R., & Noll, J. (2021). Do scaling agile frameworks address global software development risks? An empirical study. Journal of Systems and Software, 171, 110823. https://doi.org/10.1016/j.jss.2020.110823 Beringer, C., Jonas, D., & Kock, A. (2013). Behavior of internal stakeholders in project portfolio management and its impact on success. International Journal of Project Management, 31(6), 830–846. https://doi.org/10.1016/j.ijproman.2012.11.006 Biedenbach, T., & Müller, R. (2012). Absorptive, innovative and adaptive capabilities and their impact on project and project portfolio performance. International Journal of Project Management, 30(5), 621–635. https://doi.org/10.1016/j.ijproman.2012.01.016 Bjarnason, E., Gislason Bern, B., & Svedberg, L. (2022). Inter-team communication in large- scale co-located software engineering: A case study. Empirical Software Engineering, 27(2). https://doi.org/10.1007/s10664-021-10027-z Blomquist, T., & Müller, R. (2006). Practices, roles, and responsibilities of middle managers in program and portfolio management. Project Management Journal, 37(1), 52–66. https://doi.org/10.1177/875697280603700105 Bosch, J., & Bosch-Sijtsema, P. M. (2011). Introducing agile customer-centered development in a legacy software product line. Software: Practice and Experience, 41(8), 871–882. https://doi.org/10.1002/spe.1063 Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 38 Cao, L., Mohan, K., Ramesh, B., & Sarkar, S. (2013). Adapting funding processes for agile IT projects: An empirical investigation. European Journal of Information Systems, 22(2), 191– 205. https://doi.org/10.1057/ejis.2012.9 Castillo-Montoya, M. (2016). Preparing for interview research : The interview protocol refinement framework. The Qualitative Report, 21(5), 811–831. Chow, T., & Cao, D.-B. (2008). A survey study of critical success factors in agile software projects. Journal of Systems and Software, 81(6), 961–971. https://doi.org/10.1016/j.jss.2007.08.020 Cockburn, A., & Highsmith, J. (2001). Agile software development: The people factor. Computer, 11, 131–133. Conboy, K. (2009). Agility from first principles: Reconstructing the concept of agility in information systems development. Information Systems Research, 20(3), 329-354,478. https://doi.org/10.1287/isre.1090.0236 Conboy, K., & Carroll, N. (2019). Implementing large-scale agile frameworks: Challenges and recommendations. IEEE Software, March/April 1–9. https://doi.org/10.1109/MS.2018.2884865 Conboy, K., Coyle, S., Wang, X., & Pikkarainen, M. (2011). People over process: Key challenges in agile development. IEEE Software, 28(4), 48–57. https://doi.org/10.1109/MS.2010.132 Cooper, R. G., Edgett, S. J., & Kleinschmidt, E. J. (1999). New product portfolio management: Practices and performance. Journal of Product Innovation Management, 16, 333–351. https://doi.org/10.1111/1540-5885.1640333 Cooper, R. G., Edgett, S. J., & Kleinschmidt, E. J. (2002). Portfolio management: Fundamental to new product success. In P. Belliveau, A. Griffin, & S. Somermeyer (Eds.), The PDMA Tool Book 1 for New Product Development (Vol. 9, pp. 331–364). John Wiley & Sons. Dietrich, P., & Lehtonen, P. (2005). Successful management of strategic intentions through multiple projects – Reflections from empirical study. International Journal of Project Management, 23(5), 386–391. https://doi.org/10.1016/j.ijproman.2005.03.002 Digital.ai. (2021). 15th state of the agile report - Agile adoption accelerates across enterprise. In Annual Report for the State of Agile. https://stateofagile.com Dikert, K., Paasivaara, M., & Lassenius, C. (2016). Challenges and success factors for large- scale agile transformations: A systematic literature review. Journal of Systems and Software, 119, 87–108. https://doi.org/10.1016/j.jss.2016.06.013 Dingsøyr, T., Fægri, T. E., & Itkonen, J. (2014). What is large in large-scale? A taxonomy of scale for agile software development. In Proceedings of the International Conference on Product- Focused Software Process Improvement (pp. 273–276). Cham, Switzerland, Springer International Publishing. Dingsøyr, T., Falessi, D., & Power, K. (2019). Agile development at scale: The next frontier. IEEE Software, 36(2), 30–38. https://doi.org/10.1109/MS.2018.2884884 Dingsøyr, T., & Moe, N. B. (2014). Towards principles of large-scale agile development: A summary of the workshop at XP2014 and a revised research agenda. International Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 39 Conference on Agile Software Development, 1–8, Rome Italy. https://doi.org/10.1007/978-3- 319-14358-3 Dingsøyr, T., Moe, N. B., Fægri, T. E., & Seim, E. A. (2017). Exploring software development at the very large-scale: A revelatory case study and research agenda for agile method adaptation. Empirical Software Engineering, 1–31. https://doi.org/10.1007/s10664-017-9524- 2 Donaldson, L. (2001). The contingency theory of organizations (1st ed.). Thousand Oaks, CA, USA, SAGE Publications Inc. Ebert, C., & Paasivaara, M. (2017). Scaling agile. IEEE Software, 34(6), 98–103. https://doi.org/10.1109/MS.2017.4121226 Edison, H., Wang, X., & Conboy, K. (2021). Comparing methods for large-scale agile software development: A systematic literature review. IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.2021.3069039 Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14(4), 532–550. https://doi.org/10.5465/amr.1989.4308385 Eisenhardt, K. M., & Graebner, M. E. (2007). Theory building from cases: Opportunities and challenges. Academy of Management Journal, 50(1), 25–32. https://doi.org/10.5465/amj.2007.24160888 Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11 Fortune, J., & White, D. (2006). Framing of project critical success factors by a systems model. International Journal of Project Management, 24(1), 53–65. https://doi.org/10.1016/j.ijproman.2005.07.004 Gandomani, T. J., Zulzalil, H., Ghani, A. A. A., Sultan, A. B. Md., & Parizi, R. M. (2015). The impact of inadequate and dysfunctional training on agile transformation process: A grounded theory study. Information and Software Technology, 57, 295–309. https://doi.org/10.1016/j.infsof.2014.05.011 Gemino, A., Horner Reich, B., & Serrador, P. M. (2021). Agile, traditional, and hybrid approaches to project success: Is hybrid a poor second choice? Project Management Journal, 52(2), 161–175. https://doi.org/10.1177/8756972820973082 Geraghty, T. (2020). A short critique of SAFe – The scaled agile framework. tomgeraghty.co.uk. https://tomgeraghty.co.uk/index.php/a-short-critique-of-safe/ Hansen, L. K., & Svejvig, P. (2022). Seven decades of project portfolio management research (1950–2019) and perspectives for the future. Project Management Journal, 53(3), 277–294. https://doi.org/10.1177/87569728221089537 Herman, M. (2017). Introducing enterprise Scrum for business agility: Scale Scrum from single teams to whole organizations. https://michaelherman.com/publications/agile/EnterpriseScrumIntro_v4.0.pdf Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 40 Hoda, R., & Murugesan, L. K. (2016). Multi-level agile project management challenges: A self- organizing team perspective. Journal of Systems and Software, 117, 245–257. https://doi.org/10.1016/j.jss.2016.02.049 Hoda, R., Noble, J., & Marshall, S. (2009). Negotiating contracts for agile projects: A practical perspective. International Conference on Agile Processes and Extreme Programming in Software Engineering, May, 186–191. https://doi.org/10.1007/978-3-319-08581-4 Hoda, R., Noble, J., & Marshall, S. (2011). The impact of inadequate customer collaboration on self-organizing agile teams. Information and Software Technology, 53(5), 521–534. https://doi.org/10.1016/j.infsof.2010.10.009 Hoda, R., Noble, J., & Marshall, S. (2013). Self-organizing roles on agile software development teams. IEEE Transactions on Software Engineering, 39(2), 422–444. https://doi.org/10.1109/TSE.2012.30 Hoffmann, D., Ahlemann, F., & Reining, S. (2020). Reconciling alignment, efficiency, and agility in IT project portfolio management: Recommendations based on a revelatory case study. International Journal of Project Management, 38(2), 124–136. https://doi.org/10.1016/j.ijproman.2020.01.004 Howell, D., Windahl, C., & Seidel, R. (2010). A project contingency framework based on uncertainty and its consequences. International Journal of Project Management, 28(3), 256– 264. https://doi.org/10.1016/j.ijproman.2009.06.002 Jonas, D. (2010). Empowering project portfolio managers: How management involvement impacts project portfolio management performance. International Journal of Project Management, 28(8), 818–831. https://doi.org/10.1016/j.ijproman.2010.07.002 Jonas, D., Kock, A., & Gemünden, H. G. (2013). Predicting project portfolio success by measuring management quality - A longitudinal study. IEEE Transactions on Engineering Management, 60(2), 215–226. https://doi.org/10.1109/tem.2012.2200041 Jørgensen, M. (2019). Relationships between project size, agile practices, and successful software development: Results and analysis. IEEE Software, 36(2), 39–43. https://doi.org/10.1109/MS.2018.2884863 Kalenda, M., Hyna, P., & Rossi, B. (2018). Scaling agile in large organizations: Practices, challenges, and success factors. Journal of Software: Evolution and Process, Online, e1954. https://doi.org/10.1002/smr.1954 Kasauli, R., Knauss, E., Horkoff, J., Liebel, G., & de Oliveira Neto, F. G. (2021). Requirements engineering challenges and practices in large-scale agile system development. Journal of Systems and Software, 172, 110851. https://doi.org/10.1016/j.jss.2020.110851 Kaufmann, C., Kock, A., & Gemünden, H. G. (2020). Emerging strategy recognition in agile portfolios. International Journal of Project Management, January. https://doi.org/10.1016/j.ijproman.2020.01.002 Kaufmann, C., Kock, A., & Gemünden, H. G. (2021). Strategic and cultural contexts of real options reasoning in innovation portfolios. Journal of Product Innovation Management, 38(3), 334–354. https://doi.org/10.1111/jpim.12566 Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 41 Kester, L., Hultink, E. J., & Griffin, A. (2014). An empirical investigation of the antecedents and outcomes of NPD portfolio success. Journal of Product Innovation Management, 31(6), 1199–1213. https://doi.org/10.1111/jpim.12183 Killen, C. P., Hunt, R. A., & Kleinschmidt, E. J. (2008a). Learning investments and organizational capabilities. International Journal of Managing Projects in Business, 1(3), 334– 351. https://doi.org/10.1108/17538370810883800 Killen, C. P., Hunt, R. A., & Kleinschmidt, E. J. (2008b). Project portfolio management for product innovation. International Journal of Quality & Reliability Management, 25(1), 24–38. https://doi.org/10.1108/02656710810843559 Killen, C. P., Jugdev, K., Drouin, N., & Petit, Y. (2012). Advancing project and portfolio management research: Applying strategic management theories. International Journal of Project Management, 30(5), 525–538. https://doi.org/10.1016/j.ijproman.2011.12.004 Kniberg, H., & Ivarsson, A. (2012). Scaling agile @ Spotify with tribes, squads, chapters & guilds (pp. 1–14). https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScaling.pdf Kock, A., & Gemünden, H. G. (2020). How entrepreneurial orientation can leverage innovation project portfolio management. R & D Management, 51(1), 40–56. https://doi.org/10.1111/radm.12423 Kock, A., Heising, W., & Gemünden, H. G. (2015). How ideation portfolio management influences front-end success. Journal of Product Innovation Management, 32(4), 539–555. https://doi.org/10.1111/jpim.12217 Kock, A., Heising, W., & Gemünden, H. G. (2016). A contingency approach on the impact of front-end success on project portfolio success. Project Management Journal, 47(2), 115–129. https://doi.org/10.1002/pmj.21575 Kock, A., Schulz, B., Kopmann, J., & Gemünden, H. G. (2020). Project portfolio management information systems’ positive influence on performance – the importance of process maturity. International Journal of Project Management, 38(4), 229–241. https://doi.org/10.1016/j.ijproman.2020.05.001 Kopmann, J., Kock, A., Killen, C. P., & Gemünden, H. G. (2015). Business case control in project portfolios—An empirical investigation of performance consequences and moderating effects. IEEE Transactions on Engineering Management, 62(4), 529–543. https://doi.org/10.1109/TEM.2015.2454437 Kopmann, J., Kock, A., Killen, C. P., & Gemünden, H. G. (2017). The role of project portfolio management in fostering both deliberate and emergent strategy. International Journal of Project Management, 35(4), 557–570. https://doi.org/10.1016/j.ijproman.2017.02.011 KPMG. (2017). Driving business performance: Project management survey. https://assets.kpmg.com/content/dam/kpmg/nz/pdf/July/projectmanagementsurvey- kpmg-nz.pdf. Krebs, J. (2008). Agile portfolio management. Remond, WA, USA, Microsoft Press. Kupiainen, E., Mäntylä, M. V, & Itkonen, J. (2015). Using metrics in agile and lean software development – A systematic literature review of industrial studies. Information and Software Technology, 62, 143–163. https://doi.org/10.1016/j.infsof.2015.02.005 Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 42 Laanti, M., & Kangas, M. (2015). Is agile portfolio management following the principles of large-scale agile? Case study in Finnish Broadcasting Company Yle. In Agile Conference, 2015. IEEE. https://doi.org/10.1109/Agile.2015.9 Lee, G., & Xia, W. (2010). Toward agile: An integrated analysis of quantitative and qualitative field data on software development agility. MIS Quarterly, 34(1), 87–114. Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Crawfordsville, IN, USA, Pearson Education. Leffingwell, D. (2018). SAFe 4.5 reference guide: Scaled agile framework for lean enterprises. Boston, MA, USA, Addison-Wesley Professional. Lerch, M., & Spieth, P. (2013). Innovation project portfolio management: A qualitative analysis. IEEE Transactions on Engineering Management, 60(1), 18–29. https://doi.org/10.1109/tem.2012.2201723 Martinsuo, M. (2013). Project portfolio management in practice and in context. International Journal of Project Management, 31(6), 794–803. https://doi.org/10.1016/j.ijproman.2012.10.013 Martinsuo, M., & Geraldi, J. (2020). Management of project portfolios: Relationships of project portfolios with their contexts. International Journal of Project Management, May 2019. https://doi.org/10.1016/j.ijproman.2020.02.002 Martinsuo, M., & Killen, C. P. (2014). Value management in project portfolios: Identifying and assessing strategic value. Project Management Journal, 45(5), 56–70. https://doi.org/10.1002/pmj.21452 Martinsuo, M., & Lehtonen, P. (2007). Role of single-project management in achieving portfolio management efficiency. International Journal of Project Management, 25(1), 56–65. https://doi.org/10.1016/j.ijproman.2006.04.002 Meskendahl, S. (2010). The influence of business strategy on project portfolio management and its success - A conceptual framework. International Journal of Project Management, 28(8), 807–817. https://doi.org/10.1016/j.ijproman.2010.06.007 Miles, M., Huberman, M., & Saldana, J. (2014). Qualitative data analysis: A method sourcebook (3rd ed.). Sage Publications, Inc. https://doi.org/10.1080/0140528790010406 Mishra, D., Mishra, A., & Ostrovska, S. (2012). Impact of physical ambiance on communication, collaboration and coordination in agile software development: An empirical evaluation. Information and Software Technology, 54(10), 1067–1078. https://doi.org/10.1016/j.infsof.2012.04.002 Misra, S. C., Kumar, V., & Kumar, U. (2009). Identifying some important success factors in adopting agile software development practices. Journal of Systems and Software, 82(11), 1869–1890. https://doi.org/10.1016/j.jss.2009.05.052 Mosavi, A. (2014). Exploring the roles of portfolio steering committees in project portfolio governance. International Journal of Project Management, 32(3), 388–399. https://doi.org/10.1016/j.ijproman.2013.07.004 Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 43 Müller, R., Martinsuo, M., & Blomquist, T. (2008). Project portfolio control and portfolio management performance in different contexts. Project Management Journal, 39(3), 28–42. https://doi.org/10.1002/pmj.20053 Niederman, F., Lechler, T., & Petit, Y. (2018). A research agenda for extending agile practices in software development and additional task domains. Project Management Journal, 49(6), 3–17. https://doi.org/10.1177/8756972818802713 OECD. (2021). Enterprises by business size (indicator). Organisation for Economic Co-operation and Development (OECD). https://doi.org/10.1787/31d5eeaf-en Olszewska, M., Heidenberg, J., Weijola, M., Mikkonen, K., & Porres, I. (2016). Quantitatively measuring a large-scale agile transformation. The Journal of Systems and Software, 117, 258. DOI: https://doi.org/10.1016/j.jss.2016.03.029 Paasivaara, M., Behm, B., Lassenius, C., & Hallikainen, M. (2018). Large-scale agile transformation at Ericsson: A case study. Empirical Software Engineering, 23(5), 2550–2596. https://doi.org/10.1007/s10664-017-9555-8 Padovani, M., & Carvalho, M. M. (2016). Integrated PPM process: Scale development and validation. International Journal of Project Management, 34(4), 627–642. https://doi.org/10.1016/j.ijproman.2016.01.006 Patanakul, P., & Milosevic, D. (2009). The effectiveness in managing a group of multiple projects: Factors of influence and measurement criteria. International Journal of Project Management, 27(3), 216–233. https://doi.org/10.1016/j.ijproman.2008.03.001 Petro, Y., & Gardiner, P. (2015). An investigation of the influence of organizational design on project portfolio success, effectiveness and business efficiency for project-based organizations. International Journal of Project Management, 33(8), 1717–1729. https://doi.org/10.1016/j.ijproman.2015.08.004 Pikkarainen, M., Salo, O., Kuusela, R., & Abrahamsson, P. (2012). Strengths and barriers behind the successful agile deployment—Insights from the three software intensive companies in Finland. Empirical Software Engineering, 17(6), 675–702. https://doi.org/10.1007/s10664-011-9185-5 Rank, J., Unger, B. N., & Gemünden, H. G. (2015). Preparedness for the future in project portfolio management: The roles of proactiveness, riskiness and willingness to cannibalize. International Journal of Project Management, 33(8), 1730–1743. https://doi.org/10.1016/j.ijproman.2015.08.002 Rauch, A., Wiklund, J., Lumpkin, G. T., & Frese, M. (2009). Entrepreneurial orientation and business performance: An assessment of past research and suggestions for the future. Entrepreneurship: Theory and Practice, 33(3), 761–787. https://doi.org/10.1111/j.1540- 6520.2009.00308.x Rautiainen, K., Schantz, J. Von, & Vähäniitty, J. (2011). Supporting scaling agile with portfolio management: Case paf.com. System Sciences (HICSS), 2011 44th Hawaii International Conference, 1–10. Rigby, D. K., Sutherland, J., & Noble, A. (2018). Agile at scale - How to go from a few teams to hundreds. Harvard Business Review, 96(3), 88–96. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 44 Saldana, J. (2016). The coding manual for qualitative researchers (3rd ed.). London, UK, SAGE Publications Inc. Sánchez, M. A. (2015). Integrating sustainability issues into project management. Journal of Cleaner Production, 96, 319–330. https://doi.org/10.1016/j.jclepro.2013.12.087 Saunders, M., & Lewis, P. (2018). Doing research in business and management: An essential guide to planning your project (2nd ed.). Harlow, UK, Pearson Education. Scaled-Agile. (2021). SAFe 5 for Lean enterprises. Scaled Agile, Inc. https://www.scaledagile.com/ Schein, E. H. (2009). The corporate culture survival guide. San Francisco, CA, John Wiley & Sons. Schwaber, K. (2004). Agile project management with Scrum (1st ed.). Richmond, WA, USA, Microsoft Press. Schwaber, K. (2014). unSAFe at any speed. Https://Kenschwaber.wordpress.com. https://kenschwaber.wordpress.com/2013/08/06/unsafe-at-any-speed/ Serrador, P., & Pinto, J. K. (2015). Does Agile work? — A quantitative analysis of agile project success. International Journal of Project Management, 33(5), 1040–1051. https://doi.org/10.1016/j.ijproman.2015.01.006 Serrador, P., & Turner, R. (2015). The relationship between project success and project efficiency. Project Management Journal, 46(1), 30–39. https://doi.org/10.1002/pmj.21468 Shameem, M., Kumar, C., Chandra, B., & Khan, A. A. (2017). Systematic review of success factors for scaling agile methods in global software development environment: A client- vendor perspective. Proceedings - 2017 24th Asia-Pacific Software Engineering Conference Workshops, APSECW 2017, 17–24, Nanjing, China. https://doi.org/10.1109/APSECW.2017.22 Shameem, M., Kumar, R. R., Kumar, C., Chandra, B., & Khan, A. A. (2018). Prioritizing challenges of agile process in distributed software development environment using analytic hierarchy process. Journal of Software: Evolution and Process, 30(11), 1–19. https://doi.org/10.1002/smr.1979 Sheffield, J., & Lemétayer, J. (2013). Factors associated with the software development agility of successful projects. International Journal of Project Management, 31(3), 459–472. https://doi.org/10.1016/j.ijproman.2012.09.011 Shenhar, A. J. (2001). One size does not fit all projects: Exploring classical contingency domains. Management Science, 47(3), 394–414. https://doi.org/10.1287/mnsc.47.3.394.9772 Shenhar, A. J., & Dvir, D. (2007). Reinventing project management: The diamond approach to successful growth and innovation. Boston, MA, USA, Harvard Business Review Press. Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A. C. (2001). Project success: A multidimensional strategic concept. Long Range Planning, 34(6), 699–725. https://doi.org/10.1016/S0024- 6301(01)00097-8 Slevin, D. P., & Covin, J. G. (1990). Juggling entrepreneurial style and organizational structure. Sloan Management Review, 31(2), 43–52. Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 45 Smite, D., Moe, N. B., Levinta, G., & Floryan, M. (2019). Spotify guilds: How to succeed with knowledge sharing in large-scale agile organizations. IEEE Software, 36(2), 51–56. https://doi.org/10.1109/MS.2018.2886178 Šmite, D., Moe, N. B., Šāblis, A., & Wohlin, C. (2017). Software teams and their knowledge networks in large-scale software development. Information and Software Technology, 86, 71–86. https://doi.org/10.1016/j.infsof.2017.01.003 Spieth, P., & Lerch, M. (2014). Augmenting innovation project portfolio management performance: The mediating effect of management perception and satisfaction. R&D Management, 44(5), 498–515. https://doi.org/10.1111/radm.12092 Stankovic, D., Nikolic, V., Djordjevic, M., & Cao, D.-B. (2013). A survey study of critical success factors in agile software projects in former Yugoslavia IT companies. Journal of Systems and Software, 86(6), 1663–1678. https://doi.org/10.1016/j.jss.2013.02.027 Stettina, C. J., & Hörz, J. (2015). Agile portfolio management: An empirical perspective on the practice in use. International Journal of Project Management, 33(1), 140–152. https://doi.org/10.1016/j.ijproman.2014.03.008 Suomalainen, T., Kuusela, R., & Tihinen, M. (2015). Continuous planning: An important aspect of agile and lean development. International Journal of Agile Systems and Management, 8(2), 132. https://doi.org/10.1504/IJASM.2015.070607 Sutherland, J. (2021). The Scrum@Scale Guide. https://www.scrumatscale.com/scrum-at-scale- guide/ Sweetman, R., & Conboy, K. (2018). Portfolios of agile projects : A complex adaptive systems’ agent perspective. Project Management Journal, 49(6), 18–38. https://doi.org/10.1177/8756972818802712 Teller, J. (2013). Portfolio risk management and its contribution to project portfolio success: An investigation of organization, process, and culture. Project Management Journal, 44(2), 36– 51. https://doi.org/10.1002/pmj.21327 Teller, J., & Kock, A. (2013). An empirical investigation on how portfolio risk management influences project portfolio success. International Journal of Project Management, 31(6), 817–829. https://doi.org/10.1016/j.ijproman.2012.11.012 Teller, J., Kock, A., & Gemünden, H. G. (2014). Risk management in project portfolios is more than managing project risks: A contingency perspective on risk management. Project Management Journal, 45(4), 67–80. https://doi.org/10.1002/pmj.21431 Teller, J., Unger, B. N., Kock, A., & Gemünden, H. G. (2012). Formalization of project portfolio management: The moderating role of project portfolio complexity. International Journal of Project Management, 30(5), 596–607. https://doi.org/10.1016/j.ijproman.2012.01.020 Thomas, J. C., & Baker, S. W. (2008). Establishing an agile portfolio to align IT investments with business needs. Agile 2008 Conference, 252–258, Toronto, ON, Canada. https://doi.org/10.1109/Agile.2008.29 Thummadi, B. V., Shiv, O., & Lyytinen, K. (2011). Enacted routines in agile and waterfall processes. Agile Conference (AGILE), 2011, 67–76. https://doi.org/10.1109/AGILE.2011.29 Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 46 Tolfo, C., Wazlawick, R. S., Ferreira, M. G. G., & Forcellini, F. A. (2010). Agile methods and organizational culture: Reflections about cultural levels. Journal of Software Maintenance and Evolution: Research and Practice, 23(6), 423–441. https://doi.org/10.1002/smr.483 Unger, B. N., Kock, A., Gemünden, H. G., & Jonas, D. (2012). Enforcing strategic fit of project portfolios by project termination: An empirical study on senior management involvement. International Journal of Project Management, 30(6), 675–685. https://doi.org/10.1016/j.ijproman.2011.12.002 Unger, B. N., Rank, J., & Gemünden, H. G. (2014). Corporate innovation culture and dimensions of project portfolio success: The moderating role of national culture. Project Management Journal, 45(6), 38–57. https://doi.org/10.1002/pmj.21458 Vahaniitty, J. (2012). Towards agile product and portfolio management. In Department of Computer Science and Engineering: Vol. Doctoral D. Aalto University. Vlietland, J., & van Vliet, H. (2015). Towards a governance framework for chains of Scrum teams. Information and Software Technology, 57, 52–65. https://doi.org/10.1016/j.infsof.2014.08.008 Voss, M. (2012). Impact of customer integration on project portfolio management and its success—Developing a conceptual framework. International Journal of Project Management, 30(5), 567–581. https://doi.org/10.1016/j.ijproman.2012.01.017 Voss, M., & Kock, A. (2013). Impact of relationship value on project portfolio success — Investigating the moderating effects of portfolio characteristics and external turbulence. International Journal of Project Management, 31(6), 847–861. https://doi.org/10.1016/j.ijproman.2012.11.005 Williams, T. (2005). Assessing and moving on from the dominant project management discourse in the light of project overruns. IEEE Transactions on Engineering Management, 52(4), 497–508. https://doi.org/10.1109/TEM.2005.856572 Yin, R. K. (2018). Case study research - Design and methods (6th ed.). Thousand Oaks, CA, USA, SAGE Publications Inc. Appendix 1 – Indicative interview questions Introductory questions About the participant 1. What are your past experiences, particularly concerning agile projects and portfolios? 2. What is your current role in the organisation? About the organisation 3. What is the nature of business in your portfolio? 4. How many projects are running at the same time in your portfolio? 6.1.1 Core questions Agile project management 5. What specific project methodology are you using in your projects? 6. What is the history of agile in your organisation? How did it happen? 7. What would I see you doing in your projects if I followed you through a typical day? 8. When/How do you consider an agile project successful? 9. What helps you to be successful in an agile project? Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 47 Project portfolio management 10. What is the PPM approach in your portfolio/organisation? 11. What are there the challenges/issues in your portfolio? 12. How do you check (measure) the progress of your portfolio? What is success in your portfolio? 13. When/How do you consider an agile portfolio to be successful? 14. How can your agile portfolio improve? Appendix 2 – A coding example Example quotes from interviews First cycle coding/ Coding color Case-A ... In an agile where you have to make it a safe environment to be able to raise these issues up and be able to get resolved quickly. If you try to sweep them under the carpet or try to resolve them in isolation, then they just get worse. Working in silos is bad [PpM]. …our senior portfolio manager has supported us whenever we want to try new things, we always get full support to do it, and that comes out of trusting as we are going to deliver and the customers will be happy… whatever I’m hearing these days even if it’s from the CIO down, they’re actually being very supportive of how they want us to be running next year … trust works both ways. We had all this trust for my managers, and so we need to repay that trust… We really encourage people being able to say what they actually think rather than shooting the messenger. If we are falling behind in a specific feature or if there are any blockers, we really encourage them to call it out as soon as possible, and I’ll be worried about it. And., I think that culture is very prevalent, I’ve seen in the last few years we have had a lot of change and turmoil within teams, but people do respect each other, people do feel comfortable to call things out – [RTE]. Safe culture (to raise issues Working in silos (barrier) Trusting teams Mutual trust Safe culture (open communication) Leader serving teams Respect culture Case-B Anyone in the organisation is allowed to challenge anything… but not in a harmful way… usually, it’s a collective decision, and it is the ecosystem of decisions that led to these [good] outcomes. You don’t point the finger at somebody and say you are an idiot, how could you get there… but, then as a management group, we acknowledge that these mistakes are inevitable and we can’t move forward without them… - [VP]. Safe environment (to raise issues) No blame culture Accepting failure Case-C It [agile mindset] is very limited. It’s a limited practice because, from my understanding, the culture is good and healthy here…but they [senior management] are very focused on the narrow, short-term view - [AC]. No mutual trust (barrier) Case-D The meaning of trust with real examples… so during the retrospectives, we talk about issues that are happening with the team with total transparency, and this is not personal. It’s about how we can trust each other in a way that we can grow and mature – [AC]. …Usually the programmers (DevOps) and developers are very resistant about it [sharing ideas]. They feel like they are sharing their IPs when they talk. So the stand-up is the best thing for this collaborative environment and the culture it brings to it… so it’s there but not 100% [referring to trust and respect] – [BA]. Trust culture – project level Agile culture (barrier) Collaborating environment Case-E …How do you create an aligned, cohesive team purpose and values? There’s the leadership element. So how does it integrate with the team; create that stability and structure, but also have the confidence and the trust that the team is working on the same thing – [AC1]? Trust culture – project level Leaders serving teams Australasian Journal of Information Systems Aghajani, Ahsan & Whiteside 2023, Vol 27, Research Article Agility Meets a Project Portfolio 48 Example quotes from interviews First cycle coding/ Coding color …Probably the servant leadership element is a massive thing because, when you mentioned it before with trust, if people aren’t trusted, and things are over- governed, and there’s too much finance scrutiny and that kind of thing, then you basically constrain people. So even if there are teams trying to work in an agile way, they will be strangled to the point of which they will go. But, if you give people the mission statement and the purpose and the intent and you show them the element of trust, then the coaching kind of flows down to people. You give them some autonomy; you will get way better outcomes – [AC2]. Mistrust (barrier) Micromanagement (barrier) Trusting teams Giving team autonomy Case-F I think everyone has to work on concert, and I think I did mention… about trust… giving freedom to teams by trusting them that they are capable of delivering. Trust has to go both ways… - [PpM]. Mutual trust Giving team autonomy Case-G We found that the teams worked really well when they knew each other personally, and that contributes to project success. That is a direct factor – [DL]. Close collaboration Personal relationship Second coding cycle (pattern coding) Trust culture Open communication culture No-blame culture Emerged theme Organisational culture Category Contextual SF Copyright: © 2023 authors. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non- commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. doi: https://doi.org/10.3127/ajis.v27i0.4113 http://creativecommons.org/licenses/by-nc/3.0/au/ 1 Introduction 2 Literature Review 2.1 Agile Scale-up: Challenges and Success Factors 2.2 Traditional Portfolios: Success Criteria and Success Factors 2.3 Agile Portfolios: The Research Problem 2.4 The Theoretical Lens and Conceptual Framework 3 Research Methodology 3.1 Case and Participant Selection Process 3.2 Data Collection 3.3 Data Coding and Analysis 3.4 Process and Data Validation 4 Analysis of Findings 4.1 Success Criteria for Agile Portfolios 4.2 Success Factors of Agile Portfolios 4.2.1 Contextual Success Factors 4.2.2 Project-level Success Factors 5 Discussion of Findings 5.1 What Constitutes Agile Portfolio Success? 5.2 What Factors Contribute to Agile Portfolio Success? 5.2.1 New Contextual Success Factors 5.2.2 New Project-level Success Factors 5.2.3 The Effect of Uncertainty and Consequences SFs 5.3 Research Implications 5.3.1 Theoretical contribution 5.3.2 Managerial Implications 5.4 Limitations 6 Conclusion Introductory questions 6.1.1 Core questions