Paper title (Paper Title style) Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 1 Machine Learning Based Decision-Making: A Sensemaking Perspective Jingqi (Celeste) Li The University of Queensland, Brisbane, Queensland, Australia Morteza Namvar The University of Queensland, Brisbane, Queensland, Australia m.namvar@business.uq.edu.au Ghiyoung P. Im University of Louisville, Louisville, Kentucky, USA Saeed Akhlaghpour The University of Queensland, Brisbane, Queensland, Australia Abstract The integration of machine learning (ML), functioning as the core of various artificial intelligence (AI)-enabled systems in organizations, comes with the assertion that ML models offer automated decisions or assist domain experts in refining their decision-making. The current research presents substantial evidence of ML’s positive impact on business and organizational performance. Nonetheless, there is a limited understanding of how decision- makers participate in the process of generating ML-driven insights and enhancing their comprehension of business environments through ML outcomes. To enhance this engagement and understanding, this study examines the interactive process between decision-makers and ML experts as they strive to comprehend an environment and gather business insights for decision-making. It builds upon Weick’s sensemaking model by integrating ML’s pivotal role. By conducting interviews with 31 ML experts and ML end-users, we explore the dimensions of sensemaking in the context of ML utilization for decision-making. Consequently, this study proposes a process model which advances the organizational ML research by operationalizing Weick’s work into a structured ML-driven sensemaking model. This model charts a pragmatic pathway, outlining the interaction sequence between decision-makers and ML tools as they navigate through recognizing and utilizing ML, exploring opportunities, assessing ML model outcomes, and translating ML models into action, thereby advancing both the theoretical framework and its practical deployment in organizational contexts. Keywords: Machine Learning (ML), decision-making, sensemaking. 1 Introduction Artificial Intelligence (AI) is reshaping the foundation of business operations, fundamentally altering how companies operate and compete, as it facilitates smart services and automates tasks traditionally carried out by humans (Cui et al., 2022). Machine Learning (ML) serves as the driving force for decision-making in a variety of AI systems (Namvar et al., 2022). By training on vast datasets, learning intricate patterns, and generating predictive models, these AI systems autonomously analyze new data, recognize trends, and make accurate decisions based on their ‘learned knowledge’. As an illustration, ML is widely used for targeting prospective customers (Simester et al., 2020), making inventory replenishment decisions (M. Li & Li, 2022), or predicting and selecting hedge fund returns (Wu et al., 2021). Decisions Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 2 stemming from AI systems, where insights are derived from applying ML to large datasets, might introduce an entirely innovative approach to solving business problems (van den Broek et al., 2021). Consequently, both practitioners and researchers have emphasized the necessity of further research on the utilization of ML in organizational decision-making procedures (Enholm et al., 2021). Unlike conventional decision support tools, ML is distinguished by its unique characteristics (Collins et al., 2021). ML systems are noted for their advanced learning capacity (van den Broek et al., 2021), self-sufficiency (Teodorescu et al., 2021), and a remarkable level of opacity (Lebovitz et al., 2021), which sets them apart even from other intelligent technologies (Baird & Maruping, 2021). These attributes contribute not only to ML’s effectiveness but also present certain challenges. For instance, the opacity of ML systems can raise issues regarding transparency and accountability in decision-making (de Laat, 2018) leading to potentially suboptimal decisions based upon various, unrecognized biases (Manyika et al., 2017). According to the McKinsey Global Institute report (2017) within AI systems, ML is being adopted for tasks ranging from data analysis to customer service, and it projects potential job displacement and creation scenarios. The report highlighted the transformative potential of ML in industries such as manufacturing, healthcare, finance, and more. The way in which AI systems and their decision support engine, ML, enhance organizational decision-making differs significantly from traditional Information Technology (IT) approaches. Conventional decision support tools rely on a set of data to provide suggestions for organizational decision- making (Chen et al., 2012). While ML systems do automate decision-making, they also iteratively optimize their approaches using additional data to enhance outcomes (Collins et al., 2021). Nonetheless, creating ML systems solely driven by a purely technical perspective can lead to ineffective use and unintended repercussions (Namvar et al., 2022) and challenges which we have already listed above. These challenges underscore the imperative for a thorough evaluation of the implications of decisions made by ML and the potential ramifications these decisions might exert on businesses and end-users. In developing and introducing these systems, ML experts should engage in collaboration with ML end-users, encompassing both decision-makers and internal and external clients (van den Broek et al., 2021). However, ML initiatives inherently entail challenges (Ligon & Sim, 2000). For example, organizational decision-making might anticipate outcomes that are impractical considering the available data, or ML experts might worry about the future uses of these systems in ways that are not aligned with their value systems. Psychological research (see f.ex. Kahneman & Klein, 2009) suggests that proficient decision-making within intricate and disputatious social contexts hinges on more than mere technical knowledge, logic, or algorithms. The existing knowledge on how ML operates at a micro, detailed, granular level in decision-making processes is limited or incomplete, underscoring the need for more in- depth research to achieve a comprehensive understanding of how ML functions in decision- making contexts (Namvar et al., 2022). This research delves into the ways human reasoning processes can enhance individuals’ sensemaking by adeptly tailoring decisions to the specific context. Given the idiosyncrasies of ML use for decision-making, our research examines how decision-makers can approach these challenging situations and therefore, this study asks, “How do managers use ML models to gain insight for their decision-making?” To address the aforementioned research question, we conducted 31 interviews with ML end users and experts and aimed to present a process model of ML-driven sensemaking, grounded Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 3 in Weick’s sensemaking model (Weick, 1995; Weick et al., 2005). Sensemaking is defined as “the process through which individuals work to understand novel, unexpected, or confusing events” (Maitlis & Christianson, 2014, p. 57). It involves structuring the process to eliminate ambiguity in decision-making (Weick, 1995) and assigning significance to past events (Boland, 2008). Weick’s (1995) model of sensemaking provides an apt theoretical foundation for this study as it elucidates the sequential activities undertaken by decision-makers when employing ML. The connection between sensemaking and decision-making is reciprocal (Maitlis & Christianson, 2014): sensemaking precedes decision-making by providing descriptions of the surroundings, while it follows decision-making when the latter generates ambiguity. This model is particularly suitable for studying ML-driven decision-making due to its focus on the interpretive cycles of understanding complex environments. It mirrors the iterative learning and application inherent in ML, making it ideal for exploring how managers interact with and derive actionable insights from ML outputs. On this background we: (1) identify the properties of sensemaking in the context of ML, and (2) develop a process model to show how sensemaking occurs in organizations using ML. The process model portrays the sensemaking process of decision-makers, from identifying business problems and opportunities to finding actionable solutions through interactions with ML experts. The process model captures theoretical constructs using a sequence of events that take place over time and focus on explaining how and why certain outcomes occur at the end of the process (Newman & Robey, 1992). This research constitutes a cutting-edge exploration by intertwining Weick’s sensemaking model with ML practices in decision sciences. By doing so, it uncovers the nuanced processes by which decision-makers interact with ML algorithms, offering a novel model for ML-driven sensemaking. This model not only delineates the cognitive and practical steps involved in integrating ML into decision-making but also sheds light on the transformative impact of ML on organizational strategy and outcomes. The insights provided are set to challenge conventional narratives and inspire a rethinking of decision-making paradigms in the age of AI and ML. The ensuing sections first present the theoretical background of the study, followed by the details of the research method and its design, as well as a description of the data collection process. Then, we present the thematic analysis of the collected data along with the proposed process model of the study. Finally, the paper concludes and elaborates on the limitations of this study and opportunities for further research. 2 Theoretical Background ML is defined as the process of extracting patterns from data through learning algorithms, which then generate models for addressing real-world issues (Jordan & Mitchell, 2015). Employing learning algorithms, like artificial neural networks, k-nearest neighbors, or a blend thereof, is integral to ML (Collins et al., 2021). Hyperparameters are calibrated to tailor these learning algorithms to the specific problem. The training of an ML model, guided by both the algorithm and the data, culminates in the creation of a model that accurately encapsulates the patterns inherent in the data. ML experts set up ML systems: they narrow their focus to a specific business problem, prepare and provide data, and train the ML algorithms. This initial setup depends on ML experts’ proficiency and influences their data selection, preparation, and optimization, based on their Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 4 understanding of the application domain in question (KhakAbi et al., 2010), in our case business organizations and their environments. A deeper grasp possessed by those engaged in the ML system’s development empowers an efficient setup of ML models (Namvar et al., 2022), amplifying the impact of ML in improved decisions (Stetson et al., 2012). Conversely, uninformed or biased engagement in the process of ML model development can inadvertently infuse flawed perspectives about the problem into the ML models (Fügener et al., 2021). The input data for ML systems reflects a restricted viewpoint of a business problem, which might yield a less precise portrayal in comparison to how decision-makers or domain experts might present it (Namvar & Cybulski, 2014). In contrast, accounting for the perspective or concerns of decision-makers can offer a more comprehensive understanding of business issues within their genuine real-world settings (Namvar et al., 2022) which is not limited to the narrow problem represented in data. To unpack the problem, next, we discuss the existing research on ML and decision-making. 2.1 ML and Decision-Making The majority of research on ML and decision-making seeks to build a theory by using a variance approach that focuses on macro-level constructs (Trieu, 2017). Most studies have addressed the value-creation aspect of big data. Typically, studies investigate the sequence of analytics investments, analytics assets and capabilities, analytics impacts, and organizational performance (Grover et al., 2018; Trieu, 2017). Although there has been significant development in ML research, studies have primarily concentrated on macro-level constructs, often overlooking the micro-level phenomena (Namvar et al., 2022). A focus on the micro-level is vital; it uncovers the individual dynamics that are critical to the broader application of ML, particularly how decision-makers interact with and interpret ML outputs. At the micro-level, some research examined the use aspect of ML. Deng and Chi (2012) identified patterns of post- adoptive use of big data in general with a focus on the problems that prevent its successful use. Dinter (2013) examined the impacts of system quality and adequate information supply on effective use. Li et al. (2013) and Surbakti et al. (2019) investigated motivational and organizational elements for analytics use. Other research has evaluated the decision-making aspect of ML. For example, Ghasemaghaei and her colleagues (2019; 2017) studied decision- making quality with respect to data use, data quality, data diagnosticity, and analytics competency. Still, other research recognized the essential role of sensemaking in the decision- making process (Sharma et al., 2014) and identified the features of visual representation that facilitate sensemaking (Baker et al., 2009). While these studies have touched on aspects of ML use, it is important to understand the decision-making processes in relation to specific ML attributes to grasp the complex cognitive mechanisms decision-makers employ when utilizing data-driven insights. By focusing on sensemaking processes, we offer such a nuanced view of how managers interpret and implement ML in decision-making. 2.2 Sensemaking for Decision-Making Sensemaking is a core organizational activity for managers looking to understand their business environment and business problems (Weick, 1995), forming mental models that depict reality and causality, thereby laying the foundation for informed decision-making (Seidel et al., 2018; Woodside, 2005). In particular, sensemaking endeavors can tackle the complexities linked to the utilization of data, and ML, for decision-making (Hasan & Gould, 2001). Weick’s (1995) sensemaking model provides a robust foundation for comprehending Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 5 how individuals navigate these processes within business settings, highlighting it as an organizational process centred on human cognition to interpret data. The mechanism connecting sensemaking and decision-making is sequential: sensemaking constructs the mental models required for comprehending complex ML data, which then guides managers in evaluating and selecting appropriate actions during decision-making (Maitlis & Christianson, 2014). Decision-making and sensemaking are discrete yet interconnected processes, where decision-making typically follows sensemaking (Maitlis & Christianson, 2014). Decision-making entails evaluating a spectrum of feasible courses of action and opting for the optimal alternative. In contrast, sensemaking pertains to how managers comprehend and construe information (Hekkala et al., 2018; Namvar et al., 2018), while decision-making primarily centers on the prospective ramifications of decisions, actions, and outcomes (Boland, 2008). Thus, informed decision-making is predicated on effective sensemaking, where the latter shapes the context within which decisions are made. Weick conceives of sensemaking as a principal organizing process, unfolding through a sequence of ecological change, enactment, selection, and retention (see Figure 1). In this process, individuals organize to interpret unclear inputs and subsequently implement this understanding into the world, enhancing its structure (Weick et al., 2005). Enactment is triggered by ecological changes and includes sensemaking activities of noticing and bracketing to change flux into order by individuals. They bring into existence a variety of structures via reciprocal exchange with other actors in the organization. In selection, the number of possible interpretations for the information is reduced. Individuals identify a plausible, tentative interpretation through mental engagement and articulation. A plausible interpretation is finalized in the process of retention. Retention reflects on experience, formulates identities, and guides future interpretation. Retention cycles back to selection and enactment. Figure 1. Weick’s Model of Sensemaking (Jennings & Greenwood, 2003 adapted from Weick, 1979, p. 132) Weick’s sensemaking model is suitable for understanding ML-driven decision-making because it aligns with the iterative and interpretive nature of ML processes. ML algorithms often resemble Weick’s notion of ecological changes in the form of new data, requiring managers to enact sensemaking through noticing and bracketing, which is converting data into structured, understandable formats. Selection happens as decision-makers reduce vast potential interpretations to plausible ones, resonating with ML’s pattern identification. Finally, Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 6 retention allows for the experience with ML to inform future interpretations and decisions, much like training and fine-tuning an ML system with new data strengthens its predictive capability as we will in detail elaborate on in the discussion section. This model in nature aptly addresses the nuances of ML by mapping the sensemaking stages to how ML insights are generated, interpreted, and applied in decision-making. 3 Research Method and Data Collection This study investigates the process of ML-driven decision-making during which managers use ML to gain insight into their problem-solving. Managers are frequently involved in sensemaking when they have business data and want to enhance their understanding of the environment. This process is necessary and essential for making quality decisions. Due to the absence of previous research on ML and sensemaking, this study conducted interviews with ML experts to assess the applicability of sensemaking in optimizing ML’s role in decision- making. Interviews are selected for their effectiveness in revealing the complex practices and cognitive processes of the professionals in the ML field, which is crucial for our study’s focus on sensemaking. Such interviews facilitate an in-depth understanding of individual perceptions and practices, aligning with the qualitative inquiry's goals in IS research (Conboy et al., 2012). Thematic analysis (Braun & Clarke, 2006) was employed to scrutinize the collected data. The data analysis followed a six-step process: becoming familiar with the data, applying initial codes, recognizing themes, reviewing and refining themes, solidifying themes, and producing the final report. We interviewed participants representing diverse organizations, with a specific focus on elucidating their individual experiences in utilizing ML for decision-making. Overall, we conducted interviews with 31 participants. Table 1 presents brief profiles of study participants, along with their backgrounds and experience (Dickson, 2015). The study participants include ML consultants, ML system developers, data analysts, and decision-makers who all are engaged with or rely on ML in their daily practices and decisions. Primarily, they originate from sizable enterprises, each encompassing over 1000 employees. Only five participants came from small companies with fewer than 100 employees. The participants are from different sectors, including finance, government, IT, banking, healthcare, retail, and education. Selecting participants from various industries and sizes enhances the research’s transferability. We utilized semi-structured interviews as a framework to facilitate open and flowing discussions, rather than adhering to a rigid sequence of prompts and responses. After an informal introduction with all involved parties, we presented complex business scenarios to the study participants, requiring substantial business data. Participants shared how they used ML to address these scenarios. Some talked about their successes, while others shared instances of failure. For instance, one of our study participants recounted a situation where she presented an ML model to stakeholders. Within minutes, she realized she had lost the audience because her presentation style did not suit them, and they did not understand the outcomes. Furthermore, the interviewees addressed a series of inquiries regarding their use of ML for comprehending business landscapes and operations. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 7 Table 1. The Background of the Study Participants Note. To preserve confidentiality and protect the identities of the individuals who participated in the interviews, pseudonyms have been assigned to all interviewees. These pseudonyms have been used consistently throughout the document to ensure anonymity while maintaining the integrity of the research findings. 4 Empirical Results 4.1 Thematic analysis The cyclical process of data collection helped us prepare the transcripts and their analysis. We dubbed concepts related to the use of ML in organizations with preliminary codes by reading Sector Pseudonym Position Retail Emily Senior delivery analyst Khin IT Service Manager Finance Ruofan Data infrastructure, Senior business analyst Madison Enterprise intelligence Dale Data analyst, enterprise intelligence Emma Total Rewards Manager, Director Information Technology Alfred Data management & integration, Director Julian Planning & financial governance, Director Daniel Data analyst Hill Managing director Jane Senior Consultant Myla Senior Consultant Nathan Marketing & Sales, Director Robert Channel technical manager Ross Data visualization company, Founder Roy Software sales manager Scott Development, General manager Shane Technology strategist Shaun Executive director Healthcare Andrew Diagnostic imaging, Manager Arnaldo Business intelligence developer Glenn Data analysis, Manager Ian Diagnostic imaging, Operations manager Kim Revenue Manager Government Clark Operations, Director Rachel Analytics & Research, Director Heetai IT architecture & strategy, Manager Edmond Analysis & financial reporting, National manager Education Jordan Senior data analyst Matt Business intelligence, Manager Jennifer Customer Services, Manager Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 8 the interview transcripts. These codes include the concepts associated with using ML in organizations for better decision-making and understanding of business environments. In total, we uncovered 56 preliminary codes. Codes with akin connotations within pertinent contexts were combined to formulate final codes. Subsequently, 24 initial concepts (common codes) were formulated, encompassing diverse facets of ML that held relevance in its application for sensemaking within organizational contexts. Figure 2 presents these concepts, along with the data structure of our analysis. For reduction and elimination, and with reflection on each transcript, we reviewed each transcript, excising statements that exhibited vagueness, repetition, or overlap. Following this, we collated statements bearing the same code to effectively showcase study participants' perspectives on distinct matters (see Table 2 for the identified first order concepts and representative quotes). Next, we organized the first-order concepts into eight coherent themes, following the methodology outlined by Gioia et al. (2013). This interpretive process involved iterative discussions among our research team to ensure a shared understanding and agreement on the thematic groupings. For instance, as depicted in Figure 2 the “integration” theme emerges through an analysis of three key first-order concepts: 1) data source, 2) business rules, and 3) business processes. Figure 2 contains the eight themes which we categorized as second-order themes and their constituting first-order concepts. These themes and their first- order concepts are explained in the following. In the remainder of the paper, IDs refer to the second column of Table 2 and are used to relate to the concepts. Integration is the consolidation of data sources found in different business processes and external databases while taking various business rules into account. Since the value of data explodes when it can be linked and fused with other data, integration is critical to recognizing the promise of ML. The back-end of ML systems is, therefore, a data integration pipeline that extracts data from distributed and usually heterogeneous sources and processes [e.g., ID3]. Integration demands a significant investment of labor, often constituting a substantial portion of the endeavor to modify business processes and ensure data quality [ID1]. It requires working with several business processes and engaging different ML stakeholders [ID2]. Recognizing is understanding what ML can do for the business. It remediates an existing fallacy or limitation [ID4], or pursues an opportunity to enhance business processes or attract customers. Our study participant Hill, highlighted that we need to consider both data opportunities and business problems when using ML. One of the major impediments to using ML in organizations is unclear requests from decision-makers at the start [e.g., ID5]. Decision- makers need to engage with ML experts to articulate requests for the extraction of business drivers [ID6]. Interactivity regards providing decision-makers with the opportunity to engage and interact with the preliminary data visualization. It can be in the form of an interactive data visualization with an option to feed various inputs and adjust the parameters. Decision- makers prefer to investigate the stories in data [ID8]. Stories assist decision-makers in exploring the available data, applying their business understanding, and making their interpretations. In complex business and data scenarios, analysts need to visualize the integrated data to engage decision-makers in finding pieces of data that require further investigation. Our study participants recognized intuitive and user-friendly [ID7] self-service interfaces as a significant effort in a substantial domain as they help decision-makers explore additional insights and know the drivers of business functions. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 9 Figure 2. The developed Concepts and Themes Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 10 Concept ID Representative Quote Data sources 1 The complexity comes when you have more than one system, just trying to integrate them. You’ve got enough data quality issues with one system, let alone two. Business rules 2 There is always some communication with the end-users, and then the next step is to understand because that conversation creates your business rules [Ruofan]. Business processes 3 We recently did some benchmarking for financial services companies and on the delivery of the report – so most of the time we were working with procurement and finance; at the end of the report, we were working with marketing [Shaun]. Business problems 4 Quite often, people define a problem very clearly, but what they expect from the analytics is completely different from what they requested [Rachel]. Opportunity 5 There was a company who engaged Oracle or SAP but did not enable the spend analytics module or reporting modules and did not train anybody for that purpose. So, it is the scope, and then it is articulating what they believe the deliverables will be from that [Shaun]. Engagement 6 The end-users are not clear at the start, which I would say most organizations are not crystal clear when they look for a solution or a service. Engagement clarifies that and allows us to make sure that by the time we get to the end, it is going to be a yes [Shaun]. Ease of use 7 An awful lot of the executives want to carry an iPad and take the information into board meetings and use it live. A lot of the work is going into how to make the reports user-friendly to operate [Robert]. Storytelling 8 We try to tell a story. We analyze the data and decide what the key things the data is telling. Then we tell them: “these are the things that are most relevant, so it depends on you now” [Madison]. Feedback 9 Data analysts start building and delivering, and then they receive feedback. They iterate and go through and eventually break the work up into bits and deliver as much as they can [Matt]. Analytical capability 10 Our end-users were saying the data was wrong, and I said: “No, the data is not wrong; you are not asking it the right question.” So, once I showed them how to ask the right question, how to tick the right boxes, they said, “yeah exactly it is” [Ian]. Data awareness 11 Data visualization helps you find questions that you didn’t know you wanted to ask. It is about a way of exploring data and finding what insights or points that decision- makers then want to feed into a more regular business intelligence system [Ross]. Details 12 We get information from the data warehouse, put it in a paper, publish it as a PDF. If it is a starred item, decision-makers review it. Otherwise, it wouldn't even get read [Matt]. Personalizati on 13 You need to be able to have the tools and be able to manipulate and explore the possibilities yourself, in some way, visually. On the other side of the coin is the ability to engage and communicate with an audience and to convey your message to that particular audience [Ross]. Cues (Insights) 14 While decision-makers recognize the business environment is unsettled, and there are always changes and external factors, it is a cliché, but they see the big picture in that. They often get very pointy and succinct information, above or below the line. It is not grayed [Shaun]. Attitude 15 Few say we are looking for a tool that will enable us to do decision-making; to be honest, I rarely hear that language, which is what they are looking for [Shaun]. Acceptance 16 We have decision-makers with some strong opinions and beliefs about what is going on. If you are going to be developing a model which is contrary to these positions, or long-held standing positions, you have to present it very plausible and easy-to- understand [Rachel]. Validation 17 You would not make a recommendation without running it past the decision-makers. Even if they have the standard implementation, they are going to do their things to it, and they have got their quirks about why they do things. So, they will always be a point of validation [Madison]. Alerts 18 Our event management system gives us alerts, notifications, and monitoring of our KPIs. It could be our sales forecast numbers, our budget’s due, or it could be from predictive analytics, which provides an alert that could happen [Robert]. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 11 Concept ID Representative Quote Options 19 Generally, decision-makers will want from a data analyst or consultant to demonstrate the options and then a recommendation [Madison]. Plausibility 20 Not all reports have to be 100 percent confident; you can still have a report which you can say there is only 30 percent confidence, and that is better than nothing at all [Heetai]. Access 21 The idea of the executive having a mini iPad on the golf course, and getting a report sent to them, analyzing them and making some decision on the report, is very real today [Robert]. Time 22 The business should wait for various pieces of data that some data supplier can provide. And there is processing time required for technology. Also, a long time is needed for data analysts to bring this data in and analyze it [Ruofan]. Justifying 23 There is a culture of discovery and justifying your position based on evidence, which is probably a bit harder than just teaching someone how to generate an invoice differently [Nathan]. Ad hoc requests 24 Over time, once decision-makers are comfortable with understanding what they want from analytics, they apply the existing functions. Then, again, they’ll come with the same problem but for a different data set [Daniel]. Table 2. The first-order Concepts Exploring is about decision-makers investigating the preliminary data visualization and drilling into interactive visuals. Decision-makers are often unaware of the available data and opportunities; working with interactive data visualization helps them leverage their understanding of available data and explore data opportunities [ID11]. This exploration helps decision-makers provide constructive feedback to ML experts about how much the preliminary data visualization [ID9] helps them pursue data opportunities or re-formulate business problems. The analytical skills and capabilities of decision-makers determine the effectiveness of their data exploration [e.g., ID10]. ML model presentation by ML experts is about providing decision-makers with the results of their analysis. From an ML expert’s point of view, information presentation stands between interactive data visualization, discussed in interactivity themes and actionable reports. While interactive data visualization aims to assist in exploring opportunities, information presentation is used to improve the validation of the ML models by decision-makers and increase their acceptance. Our study participants cautioned about saturating decision-makers in busy business environment settings with a lot of information from ML [ID12]. They underscored the necessity for concise and focused ML models that offer optimal choices to decision-makers at the board level, customized to suit the intended audience [ID13]. The study participants expanded on the significance of simplicity and trends in ML outcomes, as decision-makers exhibit a preference for identifying anomalies above or below trend lines in charts [ID14]. Assessing is about investigating the validity of the presented ML models so that trust in using them can be developed. Decision-makers are sceptical about ML outcomes when the outcomes are not validated [ID17]. This scepticism creates even more resistance toward reports with recommended solutions for decision-makers [ID15]. For decisions at the operational level, ML can usually provide some straightforward numbers about business processes. However, at the strategic level, more effort is required to justify ML outcomes as decision-makers have strong opinions, beliefs and long-held standing positions [ID16]. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 12 Actionable ML models provide a validated analysis in the form of options and recommendations from which the final business decision can be made, and actions taken. Although intensive work on ML can be ongoing, our study participants’ repeated complaints about ML concerned its effective and sustainable use. This theme reflects a demand for actionable ML and involves several features that enable the significant use of ML in an organization. This theme identifies four features that support the use of analytical reports in action, namely, alerts, options, plausibility, and access. Timely alerts are generated via automated reports and notify decision-makers of business events [ID18]. A variety of platforms are used to generate timely automated reports [ID21]. Reports are more effective when they are available on portable devices when decision-makers need them. Decision- makers further embrace and engage with ML models that present plausible outcomes [ID20] and imprecise alternatives [ID19]. For instance, the statistical examination of predictive models can aid decision-makers in gauging the model’s efficacy with a certain degree of certainty. Such ML models afford decision-makers a chance to exercise their discernment. Turning into action is about using validated reports in practice and applying them to business decisions. Our study participants comprehended that decision-making is intertwined with risks and unstructured scenarios and that the ambiguity of outcomes significantly impacts fact-based decision-making. They highlighted that in certain instances, intuitive decision- makers seek validation in ML models to justify preexisting notions. [ID23]. They discussed the lengthy process of generating ML models [ID22] and cautioned against using aged reports. Shaun also added that the decision-making process is not nimble, contributing to report aging. 4.2 The Process of ML-Driven Sensemaking Our analysis has identified eight themes related to using ML in organizations, but so far considered each of them only individually. To further advance the understanding of the use of ML in sensemaking, we now move beyond the individual themes to examine how individual actors and the themes related across the process of sensemaking. Figure 3 shows the cyclic, infused by feedback loops, process of ML-driven sensemaking as a sequence of encounters and episodes that over time that leads to turning the outcome of the process into action and to applying it in business decision-making. In using the concepts of encounter and episode to underline the processual nature of the sensemaking phenomenon we follow Newman and Robey (1992, p. 253) who explain that “an episode refers to a set of events that stand apart from others, thus signifying the end of one sequence of activities and the beginning of another … encounters mark beginnings and ends of episodes.” We recognize the four themes recognizing, exploring, assessing, and turning into action as the episodes and the four themes integration, interactivity, ML model presentation, and actionable ML models as encounters and thus operationalize and further extend Weick’s model of sensemaking to the context of ML. Integration presents the first encounter in the proposed process model. ML experts typically start an ML project by integrating various data sources from different business processes. Recognizing them is the first episode triggered by integration and is about understanding what ML can do for a business (Woerner & Wixom, 2015). The engagement between decision- makers and ML experts is very important in this episode and ML experts assist decision- makers so that they can understand data sources and opportunities [ID5]. Failure in this engagement limits the use of ML in everyday operational decisions (Günther et al., 2017; Torres et al., 2018). Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 13 Figure 3. Process Model of ML-Driven Sensemaking Exploring is the second episode in the process model. An interactivity encounter facilitates the transition from recognizing to exploring. ML experts develop interactive interfaces to help decision-makers explore available data. Visualized data is available via maps, charts, and diagrams and empowers decision-makers to perceive important patterns in data and to identify areas for further examination. The manner in which decision-makers perceive and engage with visualizations can markedly shape their exploration and comprehension of the accessible data. The next episode is assessing the presented information. Decision-makers are often sceptical about ML outcomes, especially when using them for strategic decision-making. They feel confident when they can effectively integrate validated insights into their decision-making (Shollo & Galliers, 2016), and make use of the potential of ML (Bhimani, 2015). An ML model presentation encounter facilitates the transition from exploring to assessing. The nature of insights unearthed from the accessible data hinges on the diversity and level of detail within the various data sources (Krishnamoorthi & Mathew, 2018; Yoo, 2015). Presenting the insights is, therefore, a challenge for ML experts. Our thematic analysis identified three concepts related to ML model presentation (i.e., cues, details, and personalization). After ML experts and decision-makers reach a shared understanding and recognize data opportunities or business problems, decision-makers start exploring the preliminary ML models. ML experts also analyze data and discover insights and cues related to the identified opportunities or problems. ML experts personalize and present the developed ML models based on the skills, capabilities, and preferences of the decision-makers [ID13]. They also need to avoid saturating decision-makers with details and highlight the star cues [ID14]. A deeper understanding of data and stronger consent is then achieved through assessing the presented information. The final episode is turning into action. The application of ML outcomes occurs after decision- makers have carefully explored and assessed the presented information and the insights. Decision-makers supplement the ML outcomes with their experiences, common sense, and contextual knowledge (Shollo & Galliers, 2016), to solve problems for which the conditions are often unknown (Shollo & Galliers, 2016). An actionable ML model encounter enables the transition from assessing to turning into action. Reports as the outcomes of ML summarize the discovered insights. Our data shows that decision-makers prefer ML models that show levels of certainty with possible options [ID19], especially in situations where the reports influence Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 14 strategic decision-making. The reports, therefore, should beneficially include recommendations and the plausibility associated with each option. The process model also comprises three feedback loops: (1) from exploring to recognizing, 2) from turning into action to exploring, and (3) from turning into action to assessing. First, during exploring, decision-makers reduce the equivocality of the data by iteratively cycling through interactive data visualizations. They improve their understanding of the available data by simplifying business-data rules and applying their experience. Along with ML experts, decision-makers then continuously refine business problems and data opportunities. Decision-makers offer feedback about their new understanding and how recognizing can be repeated and improved. Next, turning into action loops back to future rounds of exploring and assessing. Depending on the trust decision-makers gain from their experience of turning ML models into action, the feedback loops can be positively or negatively reinforcing. When decision-makers have developed trust through validation of reports [ID17], the volume and elaboration of ML outcomes in their business decision will increase. The opposite is true when they have not built trust, and they continue to seek new data sets or methods to deal with business problems. A key point of the proposed process model is that turning into action is always based on the earlier exploring [ID11] and assessing [ID17]. It also has a lagged effect on the next episode of recognizing as the engagement of end-users provides clarity for ML experts, resulting in their understanding of the business problem [ID6]. Our process model provides a decision-making model based on and in the context of applying ML systems. It demonstrates the interplay between decision-makers and ML experts. First, the model recognizes the role of decision-makers in four episodes, i.e., recognizing, exploring, assessing and turning into action. In particular, it introduces the crucial role of decision- makers in recognizing what ML can do for a business and elaborates on the capabilities of decision-makers in identifying business problems and opportunities that can be addressed by ML. Decision-makers’ preconceptions and business understanding assist in creating some variation among the business patterns discovered by ML experts from ML technologies. Decision-makers structure the business-data environment and enact ML environments by participating in the exploring episode. Next, the process model shows how ML experts contribute to the process of ML-driven sensemaking. They initiate this process through the integration encounter. Also, they play a crucial role in triggering the transition between episodes through three encounters (i.e. interactivity, ML model presentation, and actionable ML models). In each of these encounters, a different ML outcome is generated based on ML experts’ understanding of the data-business environment and the input provided by the decision-makers during the progression of the sensemaking process. 5 Discussion In this study, we applied Weick's sensemaking model to explore how ML is used in organizational decision-making. Our analysis of interviews with ML experts and decision- makers led to a process model that illustrates the interplay of various elements in ML-driven sensemaking. This model includes integration of ML, recognition of patterns, interactivity in data exploration, presentation of ML models, assessment of ML outputs, and the transformation of insights into actionable decisions. These findings offer theoretical advancements in understanding the cognitive aspects of ML in organizations. In the Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 15 subsequent sections we discuss the theoretical and practical implications, providing insights for scholars and practitioners on effectively implementing ML in decision-making processes. 5.1 Theoretical Implications The results of this study have several theoretical implications for AI-driven decision science. First, the study extends ML research by explaining how decision-makers actually use ML models for their decision-making. Past research has focused on the economic and social values of ML at a macro level. A few studies have examined the use and decision-making aspects of ML (Namvar et al., 2022). As such, the understanding of ML use at the micro-level is rather limited. This study contributes to such understanding by elucidating the actual process of ML use and its roles in decision-making. Drawing on the sensemaking theory by Weick (1995), we identified eight themes via thematic analyses and developed a process model of ML-driven sensemaking. Our process model of data-driven sensemaking (Figure 3) enriches Weick’s process model of organized sensemaking by identifying the crucial role of ML and ML experts during the process of sensemaking. To further elucidate the alignment between our identified themes and Weick’s sensemaking model, we can draw direct parallels between the stages of ecological change, enactment, selection, and retention within Weick’s model, and the themes identified through our thematic analysis. Each theme represents an element of the ML sensemaking process, reflecting the corresponding stages in Weick’s model, thus providing a theoretical grounding for our findings. As Figure 1 shows, in Weick’s model sensemaking is triggered by ecological change in the organizational environment. Similar to this, our process model is triggered by recognizing what data and ML can do for decision-making. Decision-makers and ML experts evaluate the significance of data in developing a strategy toward either mitigating the risk or pursuing opportunities associated with these changes. In the enactment phase, decision-makers enact a more ordered environment to understand the meaning of the ecological changes, and to examine more, they isolate and bracket some portions of the ecological changes (Weick, 2005). Our model suggests that enactment in the ML context can be effectively accomplished by decision-makers exploring the available data. Decision-makers attend to a certain portion of data provided in an interactive format by ML experts. They selectively filter data that represent the bracketed changes, and look for relationships. Selection is made by obtaining a tentative, plausible interpretation of the information from multiple possible meanings (Weick, 1995). Our model indicates that selection can be accomplished by assessing the presented ML models. In the assessing episode, decision-makers validate the presented ML models by ML experts and select reasonable interpretations among various possible relationship structures. Similar to retention in Weick’s model, our model recognizes retention in the form of turning into action. Decision-makers retain validated ML models and apply them to business practices. Validated ML models guide their current actions and future interpretations. In our process model the resulting ML models serve as a map of relationships between events and actions that guide decisions. Similar to the loops in Weick’s model, our model has the loopback from turning into action to extra rounds of exploring and assessing. After decision-makers select some provided reports for use in action, they may recognize the need for additional data and methods for further exploration and better assessment of their decision-making situation. Last, our study demonstrates that ML may not only support decision-making but also sensemaking. Our process model identifies a sequence of episodes and encounters of Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 16 sensemaking as interchanges between decision-makers and ML experts that take place over time before decision-makers select actionable solutions. This last finding suggests that decision-makers should recognize both, the beneficial role of such interchanges with ML experts and the significant role of ML tools, and consciously and actively engage with both to reach any actionable and favorable solutions. 5.2 Practical Implications Our study also provides several other practical implications. First, beyond for decision-making proper, decision-makers may also advantageously use ML technologies to improve their general understanding of the business environment they are acting in. To this end, decision- makers should willfully engage in the sensemaking episodes, recognizing what ML can do, exploring preliminary ML models, assessing ML outcomes, and turning ML outcomes into action, and collaborate where necessary with ML experts in the encounters which mark the beginnings and ends of the sensemaking episodes to educate ML experts about the business environment to improve the latter’s understanding of the available business data. In particular, decision-makers should embrace the provided interactivity and self-service features of ML tools and consider them an opportunity to enhance their business understanding among other through customization of the reports that are created and made available. Second, ML experts should collaborate with decision-makers and engage with them during ML model development. They should not only think of ML as a solution for an existing business problem but also consider business opportunities that can be identified from the available data. ML experts should offer preliminary ML models to decision-makers in an easy- to-use format and engage decision-makers in their exploration so that the latter can discover further opportunities and, in this context, new data needs. ML results should be presented in a way that facilitates easy assessment and subsequently improves decision-makers' trust in using ML outcomes. We also suggest practice-relevant guidance for integrating ML-driven sensemaking into organizational processes. Decision-makers makers may incorporate ML insights into strategic planning sessions, ensuring that sensemaking outcomes are translated into actionable strategies. Additionally, ML experts and decision-makers should jointly conduct periodic review meetings to assess the ongoing effectiveness of ML applications in decision-making and adjust approaches as necessary. Decision-makers could conduct regular knowledge- sharing workshops with ML experts, create comprehensive documentation on business processes, and involve them in strategic meetings for a practical understanding of business challenges and objectives. These strategies would ensure that ML experts gain an in-depth understanding of the business environment, enhancing the development of relevant and impactful ML solutions. To put these recommendations into practice, for instance, in healthcare, our findings can guide clinicians in their sensemaking processes when they utilize ML systems for decision-making, particularly in interpreting complex patient data to improve treatment outcomes and in the finance industry, the insights could assist risk officers in understanding the nuances of ML models for credit risk assessment, enhancing their ability to make sense of financial predictions and data. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 17 6 Conclusion This study aimed to understand how managers use ML models to gain insights for their decision-making. To accomplish this goal, we identified the properties of sensemaking and developed a process model to explain how sensemaking takes place in organizations that use ML models for decision-making. Our research is a response to calls for a micro-level understanding of the sensemaking phenomenon associated with ML. To achieve this objective, we conducted interviews with 31 study participants, to assess both the strengths and limitations of ML within the context of the sensemaking process These were reorganized through the derivation of a ML-driven sensemaking process model which includes the decision-maker episodes of recognizing, exploring, assessing, and turning into action and the MLF expert encounters of integration, interactivity, ML model presentation and actionable ML models. Our study provides valuable insights but also acknowledges certain limitations, which pave the way for prospective research. Similar to any qualitative study, this research design does not ascertain additional interdependent factors that impact ML productivity, even though several of these were explored during the conducted interviews. Notably, we did not investigate decision-makers’ degree of domain knowledge and experience, nor how this affects their interactions with ML. Additionally, ML experts might introduce challenges or biases during the sensemaking process. Future research could undertake a detailed exploration of these elements to deepen our understanding of ML in decision-making. For instance, conducting case studies to examine how decision-makers’ domain knowledge improves their sensemaking within specific organizational settings. Investigations might also assess the conditions conducive to generating insights via ML and the impact of outsourcing ML functions, which introduces additional challenges such as the need for frequent knowledge updates. Future research can also open a discussion on the variability of AI experts’ involvement in business contexts and how this variability affects the effectiveness of AI model optimization. While our proposed model draws from a diverse array of industries and suggests broad applicability, we recognize that unique characteristics of different industries and organizational contexts may affect the model's generalizability and transferability. Thus, we invite further studies to validate and test our model across various settings. Moreover, while our study is anchored in Weick’s sensemaking model, an integration of perspectives from other disciplines could yield additional theoretical richness. Future studies might employ agency theory, stakeholder theory, or concepts from cognitive psychology and human- computer interaction to provide a more comprehensive understanding of the nuances involved in ML and decision-making. Finally, in our study, we focused on interviews for data collection. While this method provided rich insights, we acknowledge the limitation of not incorporating other methods such as document analysis or quantitative approaches. These methods could offer different perspectives and enhance the robustness of the findings. Future research could benefit from employing a mixed-methods approach, which would allow for a more comprehensive analysis and strengthen the overall research design. Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 18 Acknowledgments We acknowledge Professor Karlheinz Kautz's significant role in the development of our manuscript. His constructive feedback and support as Editor-in-Chief have been instrumental in the evolution and success of our research. We also extend our thanks to the review team for their valuable input. References Baird, A., & Maruping, L. M. (2021). The next generation of research on IS use: A theoretical framework of delegation to and from agentic IS artifacts. MIS Quarterly, 45(1), 315–341. https://doi.org/10.25300/MISQ/2021/15882 Baker, J., Jones, D., & Burkman, J. (2009). Using visual representations of data to enhance sensemaking in data exploration tasks. Journal of the Association for Information Systems, 10, 533–559. https://doi.org/10.17705/1jais.00204 Bhimani, A. (2015). Exploring big data’s strategic consequences. Journal of Information Technology, 30(1), 66–69. https://doi.org/10.1057/jit.2014.29 Boland, R. (2008). Decision making and sensemaking. In Handbook on Decision Support Systems (pp. 55–63). Berlin, Heidelberg, Germany: Springer Publishing. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS Quarterly, 36(4), 1165–1188. https://doi.org/10.2307/41703503 Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60, Article 102383. https://doi.org/10.1016/j.ijinfomgt.2021.102383 Conboy, K., Fitzgerald, G., & Mathiassen, L. (2012). Qualitative methods research in information systems: Motivations, themes, and contributions. European Journal of Information Systems, 21(2), 113–118. https://doi.org/10.1057/ejis.2011.57 Cui, R., Li, M., & Zhang, S. (2022). AI and procurement. Manufacturing & Service Operations Management, 24(2), 691–706. https://doi.org/10.1287/msom.2021.0989 de Laat, P. B. (2018). Algorithmic decision-making based on machine learning from big data: Can transparency restore accountability? Philosophy & Technology, 31(4), 525–541. https://doi.org/10.1007/s13347-017-0293-z Deng, X., & Chi, L. (2012). Understanding postadoptive behaviors in information systems use: A longitudinal analysis of system use problems in the business intelligence context. Journal of Management Information Systems, 29(3), 291–326. https://doi.org/10.2753/MIS0742-1222290309 Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 19 Dickson, S. A. (2015). Stuck in the middle: Generation Xers lived experiences in the workforce while navigating between baby boomers and millennials, and perceived impact on their job satisfaction. [Doctoral Dissertation, Colorado State University]. Mountain Scholar. https://api.mountainscholar.org/server/api/core/bitstreams/27b6c9fd-3f9e-4bd9-8a14- bb402d7d6d0f/content Dinter, B. (2013). Success factors for information logistics strategy—An empirical investigation. Decision Support Systems, 54(3), 1207–1218. https://doi.org/10.1016/j.dss.2012.09.001 Enholm, I. M., Papagiannidis, E., Mikalef, P., & Krogstie, J. (2021). Artificial intelligence and business value: A literature review. Information Systems Frontiers. 24(5), 1709–1734. https://doi.org/10.1007/s10796-021-10186-w Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021). Will humans-in-the-loop become borgs? merits and pitfalls of working with AI. MIS Quarterly, 45(3), 1527–1556. https://doi.org/10.25300/MISQ/2021/16553 Ghasemaghaei, M. (2019). Does data analytics use improve firm decision making quality? The role of knowledge sharing and data analytics competency. Decision Support Systems. 120, 14–24. https://doi.org/10.1016/j.dss.2019.03.004 Ghasemaghaei, M., Ebrahimi, S., & Hassanein, K. (2017). Data analytics competency for improving firm decision making performance. The Journal of Strategic Information Systems, 27(1), 101–113. https://doi.org/10.1016/j.jsis.2017.10.001 Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013). Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational Research Methods, 16(1), 15–31. https://doi.org/10.1177/1094428112452151 Grover, V., Chiang, R. H. L., Liang, T.-P., & Zhang, D. (2018). Creating strategic business value from big data analytics: A research framework. Journal of Management Information Systems, 35(2), 388–423. https://doi.org/10.1080/07421222.2018.1451951 Günther, W. A., Mehrizi, M. H. R., Huysman, M., & Feldberg, F. (2017). Debating big data: A literature review on realizing value from big data. The Journal of Strategic Information Systems, 26(3), 191–209. https://doi.org/10.1016/j.jsis.2017.07.003 Hasan, H., & Gould, E. (2001). Support for the sense-making activity of managers. Decision Support Systems, 31(1), 71–86. https://doi.org/10.1016/S0167-9236(00)00120-2 Hekkala, R., Stein, M.-K., & Rossi, M. (2018). Metaphors in managerial and employee sensemaking in an information systems project. Information Systems Journal, 28(1), 142– 174. https://doi.org/10.1111/isj.12133 Jennings, P. D., & Greenwood, R. (2003). Constructing the iron cage: Institutional theory and enactment. In R. Westwood & S. Clegg (Eds.), Debating organization: Point-counterpoint in organization studies (pp.195–207), Hoboken, NJ, USA: Wiley-Blackwell. Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. https://doi.org/10.1126/science.aaa8415 Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755 Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 20 KhakAbi, S., Gholamian, M. R., & Namvar, M. (2010, January 27–29). Data Mining Applications in Customer Churn Management. 2010 International Conference on Intelligent Systems, Modelling and Simulation. Liverpool, United Kingdom. https://doi.org/10.1109/ISMS.2010.49 Krishnamoorthi, S., & Mathew, S. K. (2018). Business analytics and business value: A comparative case study. Information & Management, 55(5), 643–666. https://doi.org/10.1016/j.im.2018.01.005 Li, M., & Li, T. (2022). AI automation and retailer regret in supply chains. Production and Operations Management, 31(1), 83–97. https://doi.org/10.1111/poms.13498 Li, X., Po-An Hsieh, J. J., & Rai, A. (2013). Motivational differences across post-acceptance information system usage behaviors: An investigation in the business intelligence systems context. Information Systems Research, 24(3), 659–682. https://doi.org/10.1287/isre.1120.0456 Ligon, J., & Sim, J.-S. (2000). Which companies are more profitable when using data mining techniques? Americas Conference on Information Systems, Long Beach, California, August 10–13, 2000. http://aisel.aisnet.org/amcis2000/381?utm_source=aisel.aisnet.org%2Famcis2000%2F381 &utm_medium=PDF&utm_campaign=PDFCoverPages. Maitlis, S., & Christianson, M. (2014). Sensemaking in Organizations: Taking Stock and Moving Forward. The Academy of Management Annals. The Academy of Management Annals, 8(1), 57–125. https://doi.org/10.1080/19416520.2014.873177 Manyika, J., Lund, S., Chui, M., Bughin, J., Woetzel, J., Batra, P., Ko, R., & Sanghvi, S. (2017). Jobs lost, jobs gained: Workforce transitions in a time of automation. https://epic.org/wp- content/uploads/foia/epic-v-ai-commission/EPIC-19-09-11-NSCAI-FOIA-20200529-5th- Production-pt4-Outside-Reports-Resources.pdf [Accessed March 13, 2022]. Namvar, M., & Cybulski, J. (2014). BI-based organizations: A sensemaking perspective. International Conference on Information Systems, Auckland, New Zealand, December 14–17. https://aisel.aisnet.org/icis2014/proceedings/ISStrategy/39. Namvar, M., Cybulski, J., Phang, C., Wee, C.-Y., & Tan, K. Y. (2018). Simplifying sensemaking: Concept, process, strengths, shortcomings, and ways forward for information systems in contemporary business environments. Australasian Journal of Information Systems, 22. https://doi.org/10.3127/ajis.v22i0.1654 Namvar, M., Intezari, A., Akhlaghpour, S., & Brienza, J. P. (2022). Beyond effective use: Integrating wise reasoning in machine learning development. International Journal of Information Management, 69, Article 102566. https://doi.org/10.1016/j.ijinfomgt.2022.102566 Newman, M., & Robey, D. (1992). A social process model of user-analyst relationships. MIS Quarterly, 16(2), 249–266. https://doi.org/10.2307/249578 Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 21 Seidel, S., Kruse, L. C., Székely, N., Gau, M., & Stieger, D. (2018). Design principles for sensemaking support systems in environmental sustainability transformations. European Journal of Information Systems, 27(2), 221–247. https://doi.org/10.1057/s41303-017-0039-0 Sharma, R., Mithas, S., & Kankanhalli, A. (2014). Transforming decision-making processes: A research agenda for understanding the impact of business analytics on organisations. European Journal of Information Systems, 23(4) 433–441. https://doi.org/10.1057/ejis.2014.17 Shollo, A., & Galliers, R. D. (2016). Towards an understanding of the role of business intelligence systems in organisational knowing. Information Systems Journal, 26(4), 339– 367. https://doi.org/10.1111/isj.12071 Simester, D., Timoshenko, A., & Zoumpoulis, S. I. (2020). Targeting prospective customers: Robustness of machine-learning methods to typical data challenges. Management Science, 66(6), 2495–2522. https://doi.org/10.1287/mnsc.2019.3308 Stetson, P. D., Bakken, S., Wrenn, J. O., & Siegler, E. L. (2012). Assessing electronic note quality using the physician documentation quality instrument (PDQI-9). Applied Clinical Informatics, 3(02), 164–174. https://doi.org/10.4338/ACI-2011-11-RA-0070 Surbakti, F. P. S., Wang, W., Indulska, M., & Sadiq, S. (2019). Factors Influencing Effective Use of Big Data: A Research Framework. Information & Management, 57(1), Article 103146. https://doi.org/10.1016/j.im.2019.02.001 Teodorescu, M. H., Morse, L., Awwad, Y., & Kane, G. C. (2021). Failures of fairness in automation require a deeper understanding of human-ML augmentation. MIS Quarterly, 45(3), 1483–1499. https://doi.org/10.25300/MISQ/2021/16535 Torres, R., Sidorova, A., & Jones, M. C. (2018). Enabling firm performance through business intelligence and analytics: A dynamic capabilities perspective. Information & Management, 55(7), 822–839. https://doi.org/10.1016/j.im.2018.03.010 Trieu, V.-H. (2017). Getting value from Business Intelligence systems: A review and research agenda. Decision Support Systems, 93, 111–124. https://doi.org/10.1016/j.dss.2016.09.019 van den Broek, E., Sergeeva, A., & Huysman, M. (2021). When the machine meets the expert: An ethnography of developing AI for hiring. MIS Quarterly, 45(3), 1557–1580. https://doi.org/10.25300/MISQ/2021/16559 Weick, K. E. (1979). The Social Psychology of Organizing. (2nd edn.). New York, USA: McGraw- Hill. Weick, K. E. (1995). Sensemaking in organizations (Vol. 3). Thousand Oaks, CA, USA: Sage Publications. Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the process of sensemaking. Organization Science, 16, 409–421. Woerner, S. L., & Wixom, B. H. (2015). Big data: Extending the business strategy toolbox. Journal of Information Technology, 30(1), 60–62. https://doi.org/10.1057/jit.2014.31 Australasian Journal of Information Systems Li et al. 2024, Vol 28, Research Article Machine learning based decision-making 22 Woodside, A. G. (2005). Opening up decision making: Making sense of entrepreneur and reseller business-to-business strategies. Journal of Business & Industrial Marketing, 20(7), 347–354. https://doi.org/10.1108/08858620510628588 Wu, W., Chen, J., Yang, Z., & Tindall, M. L. (2021). A cross-sectional machine learning approach for hedge fund return prediction and selection. Management Science, 67(7), 4577–4601. https://doi.org/10.1287/mnsc.2020.3696 Yoo, Y. (2015). It is not about size: A further thought on big data. Journal of Information Technology, 30(1), 63–65. https://doi.org/10.1057/jit.2014.30 Copyright: © 2024 authors. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non- commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. doi: https://doi.org/10.3127/ajis.v28.4781 http://creativecommons.org/licenses/by-nc/3.0/au/ https://doi.org/10.3127/ajis.v28.4781 1 Introduction 2 Theoretical Background 2.1 ML and Decision-Making 2.2 Sensemaking for Decision-Making 3 Research Method and Data Collection 4 Empirical Results 4.1 Thematic analysis 4.2 The Process of ML-Driven Sensemaking 5 Discussion 5.1 Theoretical Implications 5.2 Practical Implications 6 Conclusion