Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6, 7846-7864 2024 Publisher: Learning Gate DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate © 2024 by the author; licensee Learning Gate * Correspondence: tmalthaqafi@uj.edu.sa A study on inventory control system for a supply chain using Markov decision processes Torky Althaqafi1* 1College of Business, University of Jeddah, Jeddah, Saudi Arabia; tmalthaqafi@uj.edu.sa (T.A.). Abstract: This paper examines the application of Markov Decision Processes (MDPs) for controlling supply chain inventories. MDPs effectively simulate decision-making problems related to uncertainty, facilitating the determination of optimal inventory policies. The MDP framework addresses various inventory management challenges, including demand fluctuations, lead times, and holding costs. The study investigates the modeling of inventory management as a Markov decision process, detailing the states, actions, and transitions within the MDP model, along with their respective advantages and disadvantages. The research employs policy and value iteration techniques to evaluate and optimize inventory management policies. The paper assesses the proposed MDP-based inventory control system through simulations that utilize supply chain data, aiming to identify optimal policies using the MDP model. A comparative analysis of the MDP approach against conventional inventory management methods is conducted to demonstrate its efficacy in reducing costs and enhancing service levels. Additionally, the paper proposes the incorporation of multiple commodities, multi-echelon supply chains, and perishability considerations into the MDP model. The findings indicate that MDPs facilitate improved optimization of inventory policies, cost reductions, and enhanced customer service, thereby emphasizing the significance of computational complexity and the necessity for accurate data. This research provides a comprehensive investigation into the role of MDPs within inventory control systems, contributing valuable insights to the field of supply chain management. Ultimately, this study lays the groundwork for advancements in MDP-based inventory control methodologies. Keywords: Computational, Complexity, Decision-making, Inventory management. 1. Introduction The management of inventory is of paramount importance in ensuring the efficiency of supply chain operations. This concept pertains to the astute administration of the movement and volume of goods throughout a supply chain, with the objective of harmonising inventory expenditures and customer service standards. Effective inventory control is a critical aspect of business operations, as it aims to ensure the availability of sufficient stock to meet customer demand while minimising the costs associated with holding inventory and the occurrence of stockouts. Managing inventory within a supply chain presents a multifaceted undertaking, owing to numerous challenges that must be navigated. These challenges encompass a range of factors, including variability in demand, uncertainties surrounding lead times, the ever-changing nature of market conditions, and the necessity to optimise inventory levels across multiple locations. Conventional approaches to inventory control often rely on fixed regulations and assumptions, which may prove suboptimal when confronted with fluid and unpredictable circumstances. Markov Decision Processes (MDPs) provide a robust framework for the representation and resolution of decision-making dilemmas in the presence of uncertainty. MDPs effectively encapsulate the dynamic nature intrinsic to inventory control decisions by accounting for the probabilistic transitions that occur between various states, as well as the corresponding rewards or costs associated with these transitions. By incorporating these elements, MDPs offer a comprehensive and rigorous 7847 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate approach to modelling and analysing inventory control problems. Through the application of MDPs in the context of inventory control within a supply chain, organisations are enabled to make informed decisions and optimise their inventory policies effectively. This research investigation explores the application of MDPs in the conceptualisation and development of an inventory control system specifically designed for the complex dynamics of a supply chain. By leveraging the inherent capabilities of MDPs, our objective is to address the intricacies and challenges associated with inventory management, thereby providing a more streamlined and effective methodology for informed decision- making. The user has provided a numerical sequence consisting of the elements 1 and 2. In the pursuit of knowledge, our research endeavours to demonstrate the significance and efficacy of MDPs in the optimisation of inventory control policies. MDPs constitute a mathematical framework that provides a formalised approach to modelling and resolving decision-making challenges in the presence of uncertainty. These concepts are particularly relevant in the domain of inventory control within the broader context of supply chains, primarily due to their inherent capacity to effectively encapsulate the dynamic nature of decision-making processes associated with inventory management. In inventory control, practitioners are tasked with making decisions regarding order quantities, reorder points, and replenishment strategies. These decisions must be made with careful consideration of the uncertainties that inevitably arise, such as demand variability and lead time fluctuations. The utilisation of conventional methodologies often relies on fixed rules or assumptions that may not adequately accommodate dynamic circumstances. MDPs present a notably refined and adaptable methodology by conscientiously accounting for the probabilistic transitions that occur between distinct states, as well as the corresponding rewards or costs associated with these transitions. MDPs provide an effective means of encapsulating the intricate dynamics of the inventory control problem within a sequential decision framework. In the context of inventory management, the decision-maker is tasked with making choices at various states, where each state signifies the inventory level and potentially encompasses other pertinent variables. These choices manifest as actions, which may involve placing an order for a specific quantity of items or modifying the replenishment strategy. The system subsequently undergoes a transition to a novel state, which is contingent upon the stochastic dynamics encompassing demand and lead time fluctuations. The problem at hand is the lack of effective communication strategies in the workplace. Managers encounter formidable challenges when endeavouring to fulfil the imperative and ideal inventory management prerequisites within manufacturing enterprises, owing to the intricate nature engendered by the sheer magnitude of transactions and the multifarious demands emanating from customers situated in disparate geographical locations. In their scholarly work, Tochukwu and Hyacinth (2015) astutely observed that the management of inventory poses a considerable challenge, as it possesses the potential to engender significant risks that can disrupt both production processes and customer satisfaction. One of the key challenges inherent in the realm of inventory management pertains to the issue of inconsistent tracking of products. This conundrum arises when there is a lack of uniformity and reliability in the process of monitoring and tracing the movement of goods within the inventory system. Furthermore, inefficiencies in inventory warehousing can also pose a significant obstacle. This predicament arises when the storage and organisation of inventory within the warehousing facilities are not optimised, leading to suboptimal utilisation of space and resources. Another pertinent issue in the domain of inventory management is the matter of inaccurate data management. There are a number of reasons why the uncertainties felt when deciding on inventory management strategies are so important. Firstly, we aren't always able to use a precise technique that takes into account all the important factors related to the issue. Secondly, the decision-making process is further complicated since accurate predictions of human conduct are not yet available. Finally, the unknowns of the future make it difficult to predict with any degree of certainty what may happen. Modern technology and years of academic research have yielded a plethora of decision support models and systems that may help businesses make better investment and inventory management choices. In order to improve inventory management prospects while simultaneously limiting uncertainties, these tools are designed to make it easier to formulate accurate judgements. Improving business effectiveness and performance is the end goal. There have been two main types of recent innovations in the field of inventory 7848 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate optimisation for manufacturing companies: quantitative and qualitative techniques. In an attempt to address these issues, several methods are used. Reasonable and intuitive approaches, like Just-in-time, are part of the qualitative side of decision-making. In contrast, mathematical techniques like decision trees, linear programming, and the use of Markov Decision Models (MDM) are used in the quantitative side. This study's main objective is to analyse how Nigerian manufacturing organisations use Markov chains and inventory management systems. This research suggests using a non-empirical and explanatory technique to handle the complexities of inventory management and its possible relationship with the Markov chain. Examining the current literature on the Markov chain and its effects on inventory management in Nigeria's industrial industry is the goal of these techniques. 1.1. Motivation There has been an increase in the frequency and complexity of humanitarian catastrophes, and the methods for inventory management need to be modified in order to accommodate these revised requirements. This research aims to provide assistance in this regard by developing a model that is capable of continually managing the inventory of help while taking into consideration the unpredictability of both demand and contribution. In addition to this, the model takes into account the possibility of supply increases brought about by the action taken by the decision maker. Some of the possible causes of these increases include aggressive marketing operations, inventory transfers from nearby facilities, and other similar factors. In spite of this, there is a cost associated with these benefits, regardless of the underlying reason. Because of this, we provide a technique for evaluating the most effective strategy for long-term storage that takes into consideration the costs associated with inventory, perishability, and the expenses that are incurred as a result of donors increasing their contributions. Overall, the purpose of this research is to increase the ability of humanitarian organisations to respond by minimising the consequences of slow-to-onset disasters, such political upheavals and droughts, on people that are already at risk. As a result, it is possible to assert that the development and implementation of the model that has been proposed are relevant to the stages of the disaster management cycle that are concerned with preparedness and mitigation, respectively. The theory of Markov chains, which was developed by Abell'an and Liu (2013), states that if a chain is allowed to continue for a sufficient length of time, it will ultimately reach the point of equilibrium, regardless of the initial value that is entered into the chain. The mathematical notion that is commonly referred to as a "Markov chain" was named after Andrey Markov, the brilliant individual who had the audacity to suggest a model for monitoring and comprehending systems that change based on predetermined probabilities. In the field of statistical modelling, the highly regarded mathematical construct known as the Markov chain may be utilised in a number of different ways. There are a number of real-world processes that make use of these applications. Some examples include the dynamics of animal populations, the development of customer lineups at airports, and the area of continually varying currency rates. 1.2. Problem Identification Manufacturing managers face significant obstacles in fulfilling the requirements for effective and optimum inventory management. These difficulties stem from the complexity created by the numerous transactions and demands from clients who are spread out over different locations. Inventory management is plagued by a variety of issues, such as inconsistent product monitoring, inefficient inventory storage, erroneous data management, inadequate documentation, product stocking, and a failure to adapt to changing and growing demand fluctuations. There are many intricate details and difficult tasks involved in supply chain management. The optimal inventory notion for inventory management advises managers to aim for the ideal stock level, which is just enough to have products on hand without going overboard or creating issues. The goal of the optimal inventory theory is the creation of effective methods for inventory management. The Markov chain and inventory management can be related because mathematics has a propensity to look for possible solutions to issues that are related to inventories. 7849 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate 1.3. Contribution of the Study Recognize them as an essential part of a humanitarian operation and employ inventory management for long-term humanitarian operations that involve donations and demands that are unpredictable. To the best of our knowledge, none of them take into consideration the simultaneous uncertainty that exists in the demand and the distribution of donations. The bulk of the works that are published on the subject of humanitarian logistics do not deal with inventory management for perishable goods, and only a tiny number of these works investigate slow-onset disasters. Consequently, this research contributes by presenting an inventory model for progressive on-set disasters that explicitly takes into account perishability and also takes into consideration the unpredictability of demand and donation. 2. Objective of the Study 1. To Study the Development of a Markov Decision Process-based inventory control system for supply chain management. 2. To study the theory of Markov chains pertains to a mathematical framework. 3. Literature Review The mathematical process known as the Markov chain involves determining the probability of future actions based on the currently known values of certain variables. The likelihood of a specific outcome can be calculated by constructing a decision tree and assessing the probabilities of future actions at each step. Mathematically, the Markov chain represents a stochastic process in which the present state depends solely on the immediate past, thereby influencing the occurrence of future events. Notably, within the context of Markov chain theory, stochastic processes impart an appealing quality to finite Markov chains, as demonstrated in a recent study by Lotfi, Mardani, and Weber (2021). The Markov chain is a widely employed model across various disciplines, including statistics, artificial intelligence, linguistics, industrial engineering, operations research, and genomics. According to the research conducted by Akhlaghi and Malkhalifeh (2019), one of the most valuable mathematical quantitative models for inventory management is the Markov chain. The application of the Markov chain has significantly influenced the field of inventory management, as evidenced by the findings of Akumu (2014). Furthermore, this mathematical model has been demonstrated to be closely associated with enhancements in inventory management efficiency. Numerous academic studies have concentrated on the Markov chain, which has been applied in various contexts to facilitate prediction, inventory management, and forecasting. These contexts include energy supply rationing, workforce planning, and inventory market analysis, among others. Grossman, Pinto, and Ramaswamy (2019) have highlighted that the Markov chain can be characterised as a continuous-time process, referred to as the continuous-time Markov chain (CTMC), in addition to its discrete-time formulation. This statistical model serves as an effective tool for modelling a diverse range of issues across multiple fields. The Markov chain's ability to integrate differential equations and its g-steps transition probability enables a comprehensive examination of dynamic systems. The following differential equation illustrates the relationship between the variables x and y: (1+x^2)dy/dx = n(1+y^2). According to Grossman et al. (2019), the Markov chain model elucidates a sequence of random events in which the probability of transitioning from one state to another is determined exclusively by the states attained in preceding events. This characteristic engenders a natural lack of memory, as the model does not retain information about events beyond the most recent occurrence. The research conducted by Lotfi et al. (2021) posits that Markov processes are particularly effective in characterising the sequential development of events. In a discrete process, transitions can occur only at specific, predetermined intervals, whereas in a continuous process, they may transpire at any moment. Within the framework of stochastic processes, the Markov chain serves as an exceptional and invaluable analytical tool. The principal objective is to estimate the transition matrix, which is derived from well- defined system states. When examining complex systems, stochastic methodologies such as the Markov Decision Process (MDP) are employed. The five primary components of this framework operate in a coherent and sequential manner. These components can be categorised as follows: Decision epochs (i), States (ii), Actions (iii), Transition probabilities (iv), and Rewards (v). The overarching goal of the 7850 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate decision-maker is to optimise the potential of the situation, thereby guiding the environment through various developmental phases. Inventory decisions can be conceptualised as occurring at regular intervals, or decision epochs, as highlighted by Afrinaldi (2020). Depending on whether these epochs are fixed or dynamic, they can be categorised as discrete or continuous, respectively. The primary objective is to identify a course of action that aligns most closely with established criteria to achieve the desired outcome. Prudently selecting activities can lead to improved financial results and maximised profit, which in turn influences the formulation of company policies aimed at enhancing the effectiveness of inventory management techniques. In their seminal work, Nwuba et al. (2020) conducted an extensive study on stochastic inventory management employing a Markov chain approach. Their research centred on a product with unpredictable demand, with the aim of determining the optimal order and reorder quantities while considering the preferences of loyal consumers. In this context, we utilise a Markov chain model to guide our inventory strategy, which incorporates two distinct order sizes (X1 and X2) and two distinct reorder levels (r1 and r2). The order size X1 is configured in accordance with a conventional replenishment strategy upon reaching the reorder level r1. Similarly, when the reorder level r2 is attained, the order represented by X2 is also triggered. This latter order is essential for promptly meeting restocking requirements. Discrete-time, which operates within a countable space, and continuous-time, which functions within a continuous space. The results of Markov's experiment suggest that, as noted by Chatys (2020), the dynamics of a system over a specified period can be characterised by an indexed collection of random or arbitrary variables {Xt} = (X0, X1, X2, X3). In the context of inventory management, consider a specific example: the demand for mobile phones during a particular week, represented by the variable Xt. It is pertinent to note that Xt follows a Poisson distribution with a mean of 1. Several academic publications indicate that employing various Markov chain models may facilitate the resolution of inventory management challenges. In a multistage manufacturing and production process, the mathematical capability to estimate the dispersion of inventory variations can be achieved by applying the Spatio-temporal Markov Chain Model (STMCM) alongside probability chain adjustment (Akhlaghi and Malkhalifeh, 2019). Here, the concept of processes is being considered. To put it simply, processes are the building blocks of any the following procedure is used to estimate a Markov chain in a typical scenario, as stated by Ahiska, Appaji, Russell, King, and Warsing (2013): • The geographical and temporal dimensions are used to produce the mathematical representation of the condition of transition probability matrices. • To increase the precision of the (STEMCM) predictions, it is necessary to define the probability chains and joint state probabilities. • Genetic algorithm that optimizes the probability weights in STEMCM using self-adaptive mutation techniques. The outcome of the aforementioned procedure, which is predicated upon data procured from the factory, has the potential to provide inventory managers with enhanced inventory plans, thereby facilitating the attainment of optimal inventory management (Althaqafi, 2023). The utilisation of the Markov chain in the realm of inventory management encompasses intricate processes and protocols aimed at addressing the challenge of predicting inventory fluctuations. This investigation is conducted within the context of a genuine inventory system, which may encompass multiple production stages within a manufacturing firm's factory. 3.1. Inventory Management In light of the significant level of uncertainty associated with the inventory process, Haung, Meng, Liu, Liu, and Huang (2022) assert that inventory management has become an integral component of modern business operations. Numerous elements must be meticulously evaluated to achieve efficient inventory management. These factors include product selection, timeliness, and temporal considerations. Charls et al., 2024 explained that the management of medical and surgical items in healthcare facilities is a multifaceted task that necessitates the collaborative efforts of physicians, ward 7851 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate nurses, pharmacists, and inventory managers to ensure the consistent availability of medical and surgical supplies. This article introduces an innovative inventory management model that draws upon Hidden Markov Models (HMMs) and infinite horizon Markov Decision Processes (MDPs) to derive optimal inventory policies. The authors' methodological contributions have been significant for operations research in healthcare. The researchers estimated the optimal number of demand states in HMMs using a systematic search and examined its interaction with infinite horizon MDPs when developing an inventory management model for non-stationary and highly volatile hospital supplies. The researchers illustrated that the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) of the HMM for the demand model excessively penalise the model's complexity, leading to increased inventory costs. Instead, we propose integrating the HMM and MDP within a simulation framework and using the expected cost of the system to determine the optimal system design. The investigators observed an optimal number of demand states in the model, beyond which there is no benefit in increasing the model's complexity. It is crucial to emphasise that different formulations of HMMs may yield varying representations of demand patterns. Therefore, a cautious approach is necessary when employing formulations that lack explicit characterisation to derive optimal policies. Additionally, it is essential to consider the potential limitations of cost estimates that arise from model formulations that are not accurately specified. This consideration is vital to ensure that the assessment of inventory management strategies is both accurate and reliable. Zhang et al. (2024) assert that the present study successfully develops and analyses the decision- making and coordination mechanisms of a closed-loop supply chain for power batteries, employing a Markov Decision Process (MDP) model. The principal contribution of this research is the development of a comprehensive modelling framework that addresses the uncertainties and dynamics present within the supply chain while also providing coordination strategies for the various decision-makers involved. This study validates the efficacy of the proposed model and illustrates the application of dynamic programming and reinforcement learning under different conditions. The results indicate that the constructed Markov Decision Process (MDP) model can significantly enhance the overall efficiency and effectiveness of the closed-loop supply chain for power batteries, offering novel perspectives and solutions for supply chain management. The findings reveal the optimal strategies that all parties in the supply chain should adopt under varying decision cycles and probability distributions, as well as methods to maximise overall efficiency through coordination mechanisms. This research presents an innovative approach to decision-making and coordination in the closed-loop supply chain for power batteries and outlines directions for future research. The results are not only theoretically significant but also serve as a valuable reference for practical supply chain management. Francesco et al. (2024) presents an innovative heuristic aimed at enhancing supply chain inventory management within a divergent two-echelon supply chain framework. The proposed method employs a deep reinforcement learning (DRL) algorithm to determine the number of batches to produce, while integrating multi-stage stochastic programming (MSP) to optimise the quantities of batches shipped to each distribution warehouse. This dual approach effectively combines the model-based capabilities of MSP for immediate logistics decisions with the simulation-optimisation features of DRL for long-term production planning. The researchers conducted a thorough evaluation of the heuristic's performance through a series of numerical experiments. In smaller settings, where optimising solutions was feasible, the DRL-based heuristic (DRLBD) demonstrated performance remarkably close to that of exact methods. In larger and more complex scenarios, where finding an optimal solution proved challenging, DRLBD consistently outperformed both the Proximal Policy Optimisation (PPO) algorithm and the (Q)-policy benchmark, showcasing its robustness. To further validate the results, the researchers performed a sensitivity analysis by varying specific demand parameters. This analysis confirmed the stability of the results across multiple value combinations and highlighted that situation with lower peak frequencies and minimal deterministic components can be more difficult to manage. From a computational perspective, DRLBD offers rapid action computation, although the DRL training phase may serve as the primary bottleneck. This efficiency enables decision-makers to utilise the heuristic for conducting what-if analyses within a practical timeframe. Overall, the results demonstrate that the proposed heuristic not only consistently outperforms conventional deep reinforcement learning 7852 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate algorithms but also significantly reduces total costs, paving the way for more effective supply chain management strategies. As previously articulated by Costantino, Gravio, Shaban, and Tronci (2015), unplanned circumstances possess the capacity to induce unpredictability regarding inventory quantity, quality, and the duration required to fulfil orders. Furthermore, challenges associated with inventory management may arise from a variety of internal variables (Karim and Nakade, 2020). Among these factors are unanticipated workloads, protocols for equipment repair and calibration, labour strikes, transportation disruptions due to vehicle breakdowns, accidents, adverse weather conditions, natural disasters, market dynamics, price alterations, inflationary price fluctuations, and inadequate supply relative to demand. It is important to note that this enumeration is not exhaustive. According to Ahiska et al. (2013), successful inventory management necessitates careful consideration at every operational level. Effective inventory management is particularly crucial for factories employing multitasking production and complex multi-stage smoothing manufacturing processes. This effectiveness not only enables the continuous production of goods but also facilitates the prompt fulfilment of online orders for both finished and unfinished products. Traditional physical-based and manual models may prove inadequate in accurately estimating or approximating inventory management due to the inherent uncertainties and delays associated with the procurement and delivery of goods (Althaqafi, 2023). Consequently, there is a pressing need for industrial sector companies to adopt innovative solutions that leverage the principles of Markov chains. According to Silbermayr and Minner (2014), inventory management software provides efficient control and oversight of the procurement of raw materials, the production process, the storage of raw materials, and their utilisation. This facilitates the execution of manufacturing processes that are both error-free and more efficient. For manufacturing firms, it is imperative to implement an effective supply chain management system to ensure the seamless integration of production operations and inventory management. Sharma and Vishwakarma (2014) assert that such a system adopts a systematic approach to the procurement, storage, and timely delivery of components, raw materials, and finished products. Consequently, it enhances both the profitability and scalability of the business. 3.2. Markov Chain and Inventory Management Researchers have examined the relationship and interaction between Markov chains and inventory management within businesses that are engaged in supply chain and inventory management challenges. According to Huang, Meng, Liu, Liu, and Huang (2022), the Markov chain serves as a fundamental mathematical model that can assist those facing difficulties in inventory management. There are numerous practical applications of the Markov chain in the business domain. One such application involves utilising the current state of equipment to predict the quantity of defective goods that will be produced on the assembly line. Researchers Lotfi, Kargar, Hoseini, Nazari, Safavi, and Weber (2021) concluded that the implementation of Markov chains enhances inventory management, thereby facilitating the prediction of the bad debt conversion rate of commercial receivables, as evidenced by their findings. Furthermore, Myers, Wallin, and Wikstrom discovered in their 2017 study that the application of Markov chains led to improvements in inventory management. Their services enable the estimation of future brand loyalty based on an existing customer pool, employing Markov chain analysis to forecast stock and option prices. This underpins their assertion. Research on the applicability of Markov chain analysis yields contradictory findings and perspectives from a variety of stakeholders. A study conducted in the field of multistate steel production by Akhlaghi and Malkhalifeh (2019) and Huang et al. (2022) examined the efficacy of the spatio- temporal Markov model regarding its ability to forecast inventory management in manufacturing processes. In the course of these investigations, variations and inventory fluctuations in steel manufacturing enterprises were taken into consideration. The research revealed that the spatiotemporal Markov model significantly outperformed steel-adaptive mutation techniques in comparison to traditional Markov chain approaches. This superiority is attributed to the former’s greater stability. In their 2014 study, Silbermayr and Minner found that Markov chains are adversely affected by certain factors. Furthermore, their research indicated that, while Markov chain analysis is a robust method for 7853 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate generating predictions, it does not account for every forecast it produces. According to Ahiska et al. (2013) and Silbermayr and Minner (2014), Markov chains possess considerable limitations, as they fail to adequately elucidate the underlying causes of most anticipated events. • The Importance of Markov Chains for Stock Control • It streamlines procedures for managing inventories. • Reliability of out-of-sample predictions is guaranteed. • Companies who have mastered the Markov chain may use this basic approach. • Compared to other complex mathematical models, this mathematical instrument helps management with forecasts to a greater extent. • Its analytical approach, which amounts to formulaically calculating the system dependability parameters, gives it the benefit of speed and precision. There is a wide variety of Markov chain models that have been published in academic journals. 1. The Markov chain is an approximation of the Markov process. 2. Net model of queueing 3. Impart a semi-Markov model. 4. Network model for stochastic automata 5. Tempored-Petri net simulation 6. Variegated Petri dishes 7. Random Petri dish 8. Poisson queueing 9. Model for evaluating performance. 10. Mini-Max Algebra and Process Algebra 3.3. Challenges Associated with the Markov Chain • Inaccurate performance estimations can arise from a multitude of factors, including challenges in lead time analysis, the reliability of sensitivity analysis, and behaviours associated with data analysis. • Design challenges necessitating problem-solving skills encompass buffer allocations, workload distributions, and topical network architectures. • In instances where organisations are incompatible with the existing Markov model, optimisation issues related to replacement policies and quality enhancement may emerge. • Concerns regarding capacity planning, management, monitoring strategies, quality control, and difficulties related to multistage and multi-allocation inventory are all intricately connected to issues of production planning and control. • Challenges in sequencing and scheduling of production tasks, as well as issues pertaining to route variety, fleet size, and the quality of maintenance schedules, are also significant. 3.4. Theoretical Framework The esteemed Russian mathematician Andrey Markov established the significant Markov Chain Theory in 1922, as noted by Abellan-Nebot, Liu, Subiron, and Shi (2012). Within the mathematical framework referred to as "Markov chains," a system is demonstrated to transition between states according to predefined probabilistic rules. Markov chain theory posits (Abellan and Liu, 2013) that, given any arbitrary initial value, the chain will converge to an equilibrium point if operated for a sufficiently extended duration. The mathematical construct known as a "Markov chain" is named in honour of the distinguished Andrey Markov, who proposed a model that could be employed to comprehend and analyse systems that evolve according to specified probabilities. This theoretical framework asserts that it is possible to make predictions about future events based on historical data. Predictions concerning phenomena such as stock market trends or weather patterns, for instance, tend to extend significantly into the future (Akumu, 2014; Ambreen & Aftab, 2016), although the accuracy of these predictions may vary. The highly regarded mathematical construct of the Markov chain has 7854 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate numerous applications in statistical modelling. Notable examples of real-world processes where these applications are relevant include animal population dynamics, customer queuing systems at airports, and the fluctuating values of currencies. The examination of stochastic models that elucidate a sequence of potential occurrences, in which the probability of each event is contingent solely on the preceding one, is referred to as Markov chain theory (Babai, Boylan, Syntetos & Ali, 2016). Probabilistic models that illuminate a series of probable events are termed Markov chains or Markov processes. This theory is characterised by the principle that the state attained in the preceding event is the sole determinant of the probability of each subsequent event. In more accessible language, this can be described as the notion that future occurrences are dependent on present conditions. The Markov chain, a robust mathematical construct, finds extensive application in statistical modelling. Numerous real-world processes demonstrate these applications, including dynamics of animal populations, customer queue formation at airports, and the fluctuating landscape of currency exchange rates. As a widely utilised concept across various disciplines, the Markov chain constitutes a cornerstone of stochastic processes and probability theory. Fields such as physics, genetics, mathematics, economics, as well as data science and machine learning, are encompassed within this discussion. Notably, this argument is substantiated by the citations of Chaipradabgiat, Jin, and Shi (2009) and Chatys (2020). The theory of information foraging is currently a prominent topic of discussion. The concept, which emerged in the early 1990s, is generally attributed to esteemed scholars Stuart Card and Peter Pirolli. Information foraging theory investigates the intricate dynamics of human behaviour by drawing parallels between the cognitive processes of Homo sapiens and the instinctual hunting patterns of various animal species. This intriguing hypothesis posits that, akin to other animals, humans possess an inherent drive to seek out and acquire new knowledge in order to more effectively address and surmount existing challenges. Research conducted by Ching, Fung, and Ng (2002) indicates that the information foraging hypothesis elucidates the natural curiosity exhibited by humans, who are perpetually in pursuit of novel learning opportunities. This inquisitiveness is attributable to the utilisation of innate foraging capabilities, which evolved to assist animals in locating sustenance. In their study, Ahmed, Hasan, Hoque, and Alam (2018) assert that information foraging theory encompasses a range of information technologies and methodologies developed through extensive research on human behaviour to fulfil diverse needs and overcome various obstacles. The mathematical inclination to explore potential solutions to inventory-related issues serves as the connection between Markov chains and inventory management (Isik, Unal, & Una, 2017). The fundamental premise of information foraging theory is the attempt to draw parallels between the strategies employed by animals to procure food in their natural environments and the analogous approaches utilised by humans to seek information. This comparison is made to illustrate how Markov chain analysis serves as a representation of how data can enhance individuals' predictive capabilities, thereby enabling them to anticipate events such as future stock price fluctuations and the likelihood of overdue bills. This discussion addresses the topic of optimum inventory management theory. The urgent necessity to efficiently handle merchants' requests led to Harris's groundbreaking concept of optimal inventory in 1913. This necessity arose from the need to accurately predict consumer demands to mitigate discontent associated with stockouts. Harris expressed significant concern regarding the substantial budget allocated to inventories. Unfortunately, it was observed that consumer requests were sometimes not met satisfactorily, resulting in product spoilage. This adverse outcome can be attributed to the procurement of excessive inventory without an adequate storage system, as highlighted by Qiu, Tan, and Xu (2017). According to the principle of optimum inventory, managers should strive for a stock level that is appropriately balanced—sufficient to meet demand without incurring unnecessary costs or complications. Optimal inventory theory focuses on establishing efficient methods for inventory management, as discussed in the research conducted by Lang, Stulz, and Walkling (1991). The objective of this concept is to adopt a more systematic approach to anticipating consumer demands and developing effective storage and replenishment strategies. The primary aim is to identify an equilibrium between over-investment in inventory and insufficient stock levels. This is achieved through the 7855 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate implementation of efficient storage management strategies, the estimation of the ideal inventory level, and the determination of reorder points and quantities. 3.5. Research Gap The theory achieves this by proposing a Markov Decision Process (MDP) model that incorporates both the direct consideration of the perishability of the provided items and the stochastic nature of needs and donations. The MDP facilitates a straightforward approach to long-term operations, enabling the presentation of a clear solution plan that is confined to the storage level. This study addresses a significant gap in the existing literature concerning the mitigation of the potentially devastating effects of long-term operations, such as epidemics, on populations that are already vulnerable. In such circumstances, it is essential to deliver perishable supplies, including food and vaccines, to those in need. The utilisation and implementation of the proposed tool is manageable for practitioners. 4. Methodology Due to the authors' proficiency in the language and its widespread availability, C# was selected for its user-friendliness and accessibility. Although C# is a proprietary language, the model could also be developed using alternative programming languages such as Python or C++. Moreover, the Value Iteration Algorithm can be readily implemented by individuals within an organisation possessing programming skills, utilising free programming languages available in the company. This algorithm is well-established and widely acknowledged within the field. In this stage of the process, it is essential to recognise that our model operates under the assumption of an infinite time horizon. Consequently, at the commencement of the humanitarian operation, when the parameters governing contribution distribution and the required commodities are established, only a single execution is necessary throughout the duration of the operation. As Puterman (1994) indicates, the resultant policy is an ideal stable policy that delineates the most effective course of action for each state at every decision epoch. Modifications to any of the parameters, such as shelf life, demand, or contribution distributions, will necessitate the formulation of a new stationary policy to accommodate the changes that have transpired during the process. Moreover, given that the model will be executed infrequently—specifically at the onset of operations and whenever the input parameters undergo significant alterations—the efficiency of the programming language selected for model construction is of minimal importance. This is due to the infrequency of model execution. 4.1. Design of Experiments To substantiate the efficacy of our proposed strategy, we have devised a small-scale experiment that specifically scrutinises inventory management practices within a blood facility. This establishment bears the significant responsibility of procuring and distributing units of vital life-sustaining fluid not only to local medical institutions but also to philanthropic endeavours. The data regarding the demand for and donations of blood packs has been generated stochastically, under the assumption that the demand for blood packs exceeds the quantity of donations. The degradation of blood packs, in conjunction with factors such as demand and donation, constitutes a pivotal occurrence that has the potential to induce alterations in the inventory level. This, in turn, facilitates the transition between various states of the system. The states under investigation in this experimental study encompass the numerical representation of blood packs within the inventory, ranging from zero to two thousand packs. The demand for blood packs conforms to a Poisson distribution, characterised by a mean value of 90 packs per week. The act of donating blood is inherently stochastic, exhibiting characteristics that align with a Poisson distribution. Specifically, the mean number of blood packs donated per week is estimated to be 60. If deemed necessary, the blood centre possesses the capability to dispatch vehicles to remote districts to stimulate donations, thereby augmenting the quantity of blood packs contributed and amassed to align with the prevailing demand. At the blood centre, there exist a total of four distinct vehicles available to facilitate the retrieval of blood packs. Regarding each vehicle, it is postulated that the quantity of blood packs gathered every week adheres to a Poisson distribution, wherein the average number of packs amassed per day is 20. Therefore, we shall consider a collection of five potential control 7856 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate actions, namely the dispatch of zero, one, two, three, or four vehicles for pick-ups. Each dispatched vehicle shall be capable of collecting blood packs at a rate of 0, 20, 40, 60, or 80 units, respectively. Thus, it can be observed that the set denoted by 𝐴 = {0, 20, 40, 60, 80} represents the collection of control actions available to conduct this particular experiment. 4.2. Quick Cost Feature The cost criteria for this particular experiment are widely acknowledged and embraced within the relevant literature. The aforementioned parameters encompass the inventory holding cost denoted as "h," the transportation costs associated with each vehicle employed for blood pack pick-ups, represented as "ta," the costs incurred for disposing of deteriorated items referred to as "dc," the projected disposal cost denoted as "E(dc)," and, lastly, the shortage costs associated with each blood pack. The cost of inventory exhibits a direct proportionality to the quantity of packets present in stock. Conversely, the expenditures related to transportation and pick-up may vary depending on the number of vehicles employed for these operations. Furthermore, it is noteworthy that there exists a modest monetary incentive associated with the utilisation of each vehicle to facilitate pick-up operations. Each individual vehicle employed in this endeavour receives a modest remuneration of $60,000 from governmental entities, with the primary objective of incentivising the act of blood donation. The cost incurred by refraining from sending a vehicle represents the opportunity cost arising from the foregone receipt of this government incentive. The calculation of the anticipated cost of disposal takes into account the projected quantity of blood packs that will deteriorate before consumption. This projection is based on the current inventory level at each decision epoch, as demonstrated by the equation. (1) The probability symbolized as P(V = i|s, a), serves as a measure of the likelihood that I blood packs will undergo deterioration before consumption. This probability is contingent upon the present inventory levels and the cost associated with disposing of each pack, denoted as p. Furthermore, the utilization of current inventory levels serves as a basis for determining the projected inadequacy of blood packs, subsequently incorporated into the computation of anticipated deficit costs. Given the current pursuit within the realm of blood center inventory management, it is strongly advised that the inventory level should not be permitted to reach a state of complete depletion. In the hypothetical scenario wherein the availability or sufficiency of blood products becomes limited, the implementation of this prudent approach assumes paramount importance to circumvent potential complications. Accordingly, as illustrated in table (1), the financial outlay of the blood center is susceptible to a significant penalty (referred to as "sc") if a demand is not satisfied. The calculation of the total expected shortage cost is demonstrated in the following equation: (2) The probability of encountering a shortage of i blood packets, denoted as P(O = i, y), is represented based on the current inventory level s. In the event of a scarcity, the cost per pack shall be represented as 𝑠𝑐. Henceforth, the ensuing equation shall denote the cost function for the particular experiment at hand, denoted as 𝑅(𝑠, 𝑎): (3) Table 1 provides a comprehensive overview of the cost structure of the experiment at hand, effectively presenting the respective values assigned to each parameter. It is of utmost significance to underscore that the costs employed in this experimental endeavor were derived from arbitrary numerical values. 7857 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate Table 1. Costs of the experiment. Transportation and pick-up expenses that might change over time Price One automobile $ 120,00.0 / automobile Two automobiles $ 85,00.0 / automobile Three automobiles $ 75,00.0 / automobile Four automobiles $ 70,00.0 / automobile Cost of retaining inventory $ 1,00.0 / bundle Price of disposal (p) $ 1000,00.0 / bundle A shortage's cost $ 1000,00 / bundle 4.3. Rates of Degradation Blood packs undergo a process of deterioration precisely 42 days after their collection, establishing their deterioration rate as deterministic. Typically, blood packs are rendered immediately employable upon their availability. In the context of continuous humanitarian aid operations, it is essential to consider that these operations frequently occur in underdeveloped countries where local collection is not feasible. Consequently, the blood packs utilised in such operations may originate from various global locations, thereby directly impacting their shelf life. To effectively address this issue, we conducted a simulation encompassing four distinct scenarios, each relating to varying lead times and their consequential impact on the shelf life of blood packs. In each scenario, it is imperative to maintain a constant set of parameters regarding demand, donation, and cost. In the initial scenario, we considered a confined assemblage, wherein the temporal duration of viability for a blood pack is established as 42 days. In the subsequent scenarios, we accounted for lead times of 21 days, 28 days, and 35 days, resulting in corresponding shelf lives of 21 days, 14 days, and 7 days for the blood packs. It is of utmost significance to acknowledge that blood packs are subject to a distinct supply chain management system commonly referred to as the cold chain or cold supply chain. Regrettably, the present study does not delve into the intricacies of this system. Furthermore, it is imperative to underscore that the lead times serve exclusively as a means to illustrate the disparities in ordering strategies contingent upon fluctuating rates of deterioration or shelf lives. Consequently, the aforementioned factors do not exert a direct influence on the fluctuations observed in inventory levels, thereby resulting in a model that fails to incorporate the temporal delay associated with lead time. 5. Results and Discussion Upon conducting a thorough examination of the model, we successfully discerned the optimal ordering procedures for the four scenarios delineated in the preceding section. A concise presentation of the findings can be observed in Figure 1 and Table 3. The actions generated by our model correspond to the tasks to be executed during each decision epoch, determined by the inventory level at the commencement of the current decision epoch. More precisely, each individual action within the aforementioned scenario aligns with a distinct numerical value denoting the number of blood packets that must be acquired. The procurement process commences at the onset of each week, taking into account the initial quantity of blood packs available in the inventory at the beginning of that week. The model we have developed provides the decision maker with the optimal quantity of blood packets to procure for every conceivable inventory level within the stock. The objective is to effectively reduce the mean expenditures associated with the administration of inventory for the humanitarian endeavour. The optimal strategy for procurement or ordering, in terms of operational efficiency, is the utilisation of a comprehensive collection of ideal actions, each corresponding to a specific inventory level. According to the experimental set comprising five potential actions, our model has formulated a prescription for an optimal ordering policy in the initial scenario. It is important to note that this scenario assumes a proposed shelf life of 42 days, as elaborated upon in section 4.3. The optimal strategy entails deploying a fleet of four vehicles in the event that the inventory level of blood packs reaches or 7858 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate falls below the threshold of 340 units. The consequence of this action will manifest in the acquisition of an average of 80 supplementary units of blood packs. To acquire an additional 60 units of blood packs, it is advised that a convoy consisting of three vehicles be dispatched when the inventory level falls within the range of 341 to 364 units. To procure an additional 40 units of blood packs, it is recommended to dispatch a pair of vehicles once the inventory level reaches the range of 365 to 390 blood packs. To acquire an additional 20 units of blood packs, it is recommended that a singular vehicle be dispatched when the inventory level falls within the range of 391 to 428 units. When the inventory level surpasses 429 blood packs, it becomes imperative to abstain from dispatching any vehicles, thereby effectively preventing the procurement of any further blood packs. In this scenario, it has been determined that the minimum weekly average cost amounts to $62.38. The decision maker ought to embrace the aforementioned ordering procedure not only to mitigate average inventory costs but also to avert the deterioration and scarcity of blood packs. In summary, Table 2 presents the optimal ordering strategies for the four proposed scenarios concerning shelf life. The aforementioned procedure elucidates the process by which the determination is made regarding the most advantageous quantity of vehicles required and the corresponding volume of blood packs to be gathered, while duly considering the existing levels of inventory. Table 2 exhibits the fluctuations in the mean inventory cost concerning alterations in the shelf life of blood packs. Table 2. Best practices for placing orders in the provided situations. Shelf life = 42 Shelf life = 21 Shelf life = 14 Shelf life = 7 Vehicles used Blood packs pick-ups Inventory level available between 4 80 0.0 – 340.0 0.0 – 87.0 0.0 – 1.0 - 3 60 341.0 – 364.0 88.0 – 122.0 2.0 – 41.0 - 2 40 365 .0– 390.0 123.0 – 148.0 42.0 – 70.0 0.0 – 5.0 1 20 391.0 – 428.0 149.0 – 180.0 71.0 – 100.0 6.0 – 32.0 0 0 429.0– 2000.0 181.0 – 2000.0 101.0 – 2000.0 33.0 – 2000.0 Costs on a weekly average $ 62,38.0 $ 97,45.0 $ 109,14.0 $ 120,85.0 As anticipated, and as delineated in Table 2 and Figure 1, it is observed that an increase in inventory levels corresponds with a decrease in the number of blood pack pick-ups required. As the probability of a shortage approaches zero, it becomes evident that the costs associated with transportation and inventory holding significantly exceed the anticipated costs of shortages. Furthermore, it is noteworthy that as the shelf lives of products diminish, there is a corresponding increase in average inventory costs. This phenomenon is primarily attributable to the fact that perishable items tend to expire at a faster rate, thereby necessitating higher disposal costs. This relationship is clearly illustrated in Table 2. Moreover, Figure 1 illustrates the inverse relationship between the number of blood packs collected and the corresponding inventory level across the four distinct scenarios examined during the experiment. The provided graphic effectively demonstrates that as the shelf life (V) increases, there is a gradual decrease in the quantity of blood packs collected. This reduction is implemented as a precautionary measure to avert potential shortages. It is important to note that the costs associated with disposing of expired items are relatively insignificant compared to the costs incurred from shortages, primarily due to the low probability of deterioration. Furthermore, it is important to note that while we consider the imposition of a substantial penalty for the scarcity of blood packs, the anticipated costs associated with the disposal of expired items increase significantly in conjunction with the likelihood of spoilage. It is crucial to recognise that as the shelf lives of blood packs decrease, there is a corresponding rise in the volume of blood packs that are discarded due to deterioration. This phenomenon occurs concurrently with an increase in inventory 7859 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate levels, resulting in heightened disposal costs and, consequently, a reduction in the overall collection of blood packs. Figure 1. Ideal procedures for collecting blood packs. It is essential to recognise that, in the context of isolated societies where the acquisition of blood packs or other perishable commodities is challenging or impractical, the depletion of supplies and the limited capacity for blood pack retrieval or procurement—due to the transient nature of these items— may indeed present a significant threat to the well-being of the community and the effectiveness of humanitarian efforts. While our model seeks to determine optimal policies by considering the minimal operation costs, it is crucial for managers to consistently remember that the primary aim of a humanitarian operation is to protect and preserve human life. Therefore, to ensure the smooth functioning of operations and mitigate potential risks, researchers must be able to identify and strategise the operational expenditures that most appropriately align with the desired objectives in the model's objective function. 5.1. Analysing Sensitivity The evaluation of the model's robustness is conducted through the utilisation of sensitivity analysis, a method that examines the impact of parameter variations on the optimal ordering strategy. Given the aforementioned assumptions regarding the spectrum of potential actions and cost parameters, as well as the specific scenario involving a 42-day lifespan for blood packs and a fixed weekly parameter of 60 blood packs for distribution through donations, we shall now proceed to scrutinise the ramifications of altering the demand distribution parameter on both the ordering policy and the objective function. In the course of our investigation, we engage in a sequence of model iterations wherein we systematically manipulate the demand distribution parameter (𝜆) within the range of 60 to 100 blood packs per week. To effectuate this particular modification, it is imperative to note that for each iteration, the aforementioned parameter undergoes an incremental augmentation of precisely five units of blood packs each week. The findings are visually presented in Figure 2, while the numerical data is organised in Tables 3 and 4. 7860 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate Figure 2. Effect of demand parameter change on the ideal average prices. Table 3. Best practices for ordering and typical expenses for weekly demand parameters ranging from 60 to 80 units of blood. 𝜆 = 60 𝜆 = 65 𝜆 = 70 𝜆 = 75 𝜆 = 80 Vehicles used Blood packs pick-ups Availability of inventory levels 4 80 0 – 115 0 – 153 0 – 191 0 – 230 0 – 269 3 60 116 – 155 154 – 193 192 – 230 231 – 265 270 – 300 2 40 156 – 197 194 – 230 231 – 262 266 – 294 301 – 326 1 20 198 – 234 231 – 266 263 – 298 295 – 330 327 – 363 0 0 235 – 2000 267 – 2000 299 – 2000 331 – 2000 364 – 2000 Average Costs/week $ 139,72 $ 126,30 $ 109,14.0 $ 100,46 $ 86,24 Table 4. Best practices for ordering and typical expenses for weekly demand parameters ranging from 85 to 100 units of blood. 𝜆 = 85 𝜆 = 90 𝜆 = 95 𝜆 = 100 vehicles used Blood packs pick-ups Inventory level available between 4 80 0 – 305 0 – 340 0 – 374 0 – 407 3 60 306 – 332 341 – 364 375 – 397 408 – 429 2 40 333 – 358 365 – 390 398 – 422 430 – 454 1 20 359 – 396 391 – 428 423 – 462 455 – 495 0 0 396 – 2000 429 – 2000 463 – 2000 495 – 2000 Average Costs/week $ 74,18 $ 62,38 $ 48.67 $ 36,85 7861 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate The tables presented, specifically Tables 3 and 4, illustrate the diverse strategies that can be employed in the collection of blood packs, taking into consideration fluctuations in demand variability. For a given value of the demand parameter (𝜆), it can be observed that the quantity of blood packs required to prevent shortages increases in direct proportion. Based on the data presented in Table 4, it is evident that when the inventory level of blood packs reaches 306 units, the number of required blood pack collections decreases from 80 to 60. This reduction in demand is significant, as the average demand stands at 85 units. Conversely, it is imperative to maintain an inventory level of 408 blood packs in order to observe a reduction in required collections when the average demand is increased to 100. It can be inferred that while there would be an increase in the price associated with the procurement of blood packs, there would be a corresponding decrease in the expenses incurred due to shortages. Furthermore, it is noteworthy that in the scenario of increased demand, there would be a reduction in the amount of product wastage and a subsequent decrease in the costs associated with disposal. Therefore, the analysis presented in Figure 2 demonstrates that the average costs of the system exhibit an inverse relationship with the mean demand (𝜆). This implies that as the value of 𝜆 increases, the corresponding costs decrease. It is of utmost significance to acknowledge that, within the context of this study, the costs associated with disposal and shortages significantly influence inventory costs, given their substantial impact on the system. Consequently, the reduction of both expense categories would result in a considerable decline in mean expenditures. Additionally, the effects resulting from alterations in the duration of the goods' shelf life are clearly illustrated in Table 2 and Figure 1. 6. Conclusion In summary, the aforementioned research paper thoroughly investigated the utilisation of Markov Decision Processes (MDPs) in the domain of inventory control as it pertains to the intricate dynamics of the supply chain. The implications of this research hold considerable significance for both academia and the industrial sector. The application of Markov Decision Processes (MDPs) in inventory control provides a robust framework for comprehending and addressing the complexities associated with inventory management within supply chains. The findings possess the potential to offer valuable guidance to practitioners in inventory control, enabling them to implement strategies that optimise their operational performance and enhance customer satisfaction effectively. Future research directions may focus on the comprehensive investigation and enhancement of the scalability and computational efficiency of techniques based on Markov Decision Processes (MDPs). Additionally, the integration of real-time data and advanced analytics could be pursued to improve the accuracy of inventory control models. Furthermore, a thorough examination of the application of Markov Decision Processes (MDPs) in various supply chain scenarios, including multi-echelon inventory systems and collaborative supply chains, has the potential to yield significant findings and solutions for enhancing inventory management on a larger scale. 6.1. Limitations The purpose of this study was to investigate the connection that exists between the Markov chain and the inventory management practices of industrial organisations. Insight and a comprehensive understanding of the Markov chain were provided by the review, which represents a novel contribution to the existing corpus of Markov chain research. Given that the Markov chain is utilised for inventory management by a limited number of manufacturing organisations, the research design employed in this study was exploratory, and the quantitative research methodology utilised data from manufacturing firms. Consequently, the study is accompanied by several limitations. 6.2. Future Directions and Scope of the Present Work Should the model be expanded to incorporate a greater number of sources of commodities, it would facilitate a more accurate representation of the challenges and requirements faced by humanitarian organisations. This expansion would enhance the likelihood that goods with varying shelf lives would arrive at the disaster site during the same determination period. In future iterations, modifications to the 7862 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate model may consider the inclusion of multiple warehouses and collaboration among groups to establish a fair distribution of donated (and/or purchased) resources and optimise total operating expenditures. Copyright: © 2024 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). References [1] Axsäter, S., (2017). Inventory Control, 2nd edition. International Series in Operations Research & Management Science. New York: Springer Science. [2] Beckmann, M. J. and Srinivasan, S. K., (2018), An (s, S) inventory system with Poisson demands and exponential lead time, OR Spectrum, 9(4), 213- 217. [3] Benita, M., Beamon, Supply Chain Design and Analysis: Models and Methods, University of Washington, Industrial Engineering, International Journal of Production Economics (2016), Vol. 55, No. 3, pp. 281-294. [4] Berman, O., and Kim, E., (2018), Stochastic inventory policies for inventory management at service facilities, Stoch. Model., 15, 695-718. [5] Berman, O., and Sapna, K. P. (2017), Inventory management at service facilities for systems with arbitrarily distributed service times, Working paper, Faculty of Management, University of Toronto. [6] Berman, O., and Sapna, K. P., (2016), Optimal control of service for facilities holding inventory, Computers and Operations Research, 28, 429-441. [7] Charls, S., Oliveira, F. S., & Pavlov, V. (n.d.). Designing a Dynamic Hospital Inventory Management System Using Markov Models. https://ssrn.com/abstract=4912303. [8] Silver, E. A., Pyke, D. F., & Peterson, R. (1998). Inventory management and production planning and scheduling. John Wiley & Sons. [9] Stranieri, F., Fadda, E., & Stella, F. (2024). Combining deep reinforcement learning and multi-stage stochastic programming to address the supply chain inventory management problem. International Journal of Production Economics, 268, 109099. https://doi.org/https://doi.org/10.1016/j.ijpe.2023.109099 [10] Kouvelis, P., Chambers, C., & Wang, H. (2006). Supply chain management research and production and operations management: Review, trends, and opportunities. Production and Operations Management, 15(3), 449-469. [11] Scarf, H. (1959). Optimal policies for a multi-echelon inventory problem. Naval Research Logistics Quarterly, 6(3), 135-153. [12] Mahmoodi, M., & Baradaran, F. (2015). Markov decision process for vendor-managed inventory in a two-echelon supply chain. International Journal of Production Economics, 170(Part B), 582-594. [13] Pasandideh, S. H. R., & Nematollahi, M. (2019). An inventory control model in a multi-echelon supply chain considering Markov decision process. International Journal of Production Research, 57(14), 4509-4531. [14] Tajbakhsh, A., Razmi, J., & Makui, A. (2015). A Markov decision process model for inventory control in a multi- echelon supply chain. International Journal of Production Research, 53(11), 3229-3242. [15] T. Althaqafi, “Environmental and Social Factors in Supplier Assessment: Fuzzy-Based Green Supplier Selection,” Sustainability, vol. 15, no. 21, 2023, doi: 10.3390/su152115643 [16] T. Althaqafi, “Cultivating Sustainable Supply Chain Practises in Electric Vehicle Manufacturing: A MCDM Approach to Assessing GSCM Performance,” World Electr. Veh. J., vol. 14, no. 10, 2023, doi: 10.3390/wevj14100290 [17] Nguyen, N., & Kaku, B. K. (2017). A Markov decision process model for inventory control in a multi-echelon supply chain considering demand uncertainty. Journal of Manufacturing Systems, 42, 9-20. [18] Söylemez, M. S., & Şen, I. (2021). A multi-objective inventory control model for a multi-echelon supply chain under demand uncertainty. Journal of Industrial Engineering International, 17(2), 593-608. [19] Chen, Z., Dong, M., & Zhang, X. (2020). Optimal inventory control policies in a supply chain with demand learning and Markov decision process. International Journal of Production Economics, 220, 107453. [20] Dehghani, E., & Babazadeh, R. (2019). A review on supply chain inventory management models: From Markov Decision Processes to machine learning techniques. Computers & Industrial Engineering, 135, 781-800. [21] Topaloglu, S. (2010). Optimal dynamic inventory control with Markov-modulated demand processes. Naval Research Logistics, 57(1), 32-49. [22] Zhang, Y., Goh, M., & Xie, M. (2012). Optimal inventory control with Markov-modulated demand and emergency replenishment. European Journal of Operational Research, 220(2), 408-416. [23] Mahmoodi, M., & Baradaran, F. (2018). Joint inventory control and pricing in a multi-item, multi-echelon supply chain under Markov demand process. International Journal of Production Economics, 195, 166-179. [24] Shah, N. H., Chen, Y., & Petruzzi, N. C. (2017). A Markov decision process approach to online inventory management with learnable demand. Manufacturing & Service Operations Management, 19(3), 421-435. [25] Diabat, A., & Said, L. A. (2019). Multi-objective optimization of an inventory control system in a multi-tier supply chain using Markov decision processes. Journal of Industrial Engineering and Management, 12(1), 119-133. [26] Gupta, R., & Dutta, P. (2013). Inventory control with Markovian demands in a two-echelon supply chain. International Journal of Production Economics, 144(2), 481-487. https://creativecommons.org/licenses/by/4.0/ https://ssrn.com/abstract=4912303 7863 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate [27] Chiu, C. Y., & Choi, T. M. (2014). Optimal order and trade credit policy for a retailer under Markovian demand. International Journal of Production Economics, 155, 271-277. [28] Sun, M., & Leng, M. (2015). An MDP-based policy improvement algorithm for periodic inventory control problem with returns. Computers & Operations Research, 53, 99-108. [29] Zhang, D. Z., He, Y., & He, Y. (2017). Inventory management in a single-stage production system with Markovian demand and lost sales. Applied Mathematical Modelling, 42, 116-134. [30] Dubois, T., Grubbström, R. W., & Holmström, J. (2018). A Markovian inventory model for a manufacturer with non- stationary demand and service level constraints. International Journal of Production Economics, 199, 76-88. [31] Mitra, S., & Webster, S. (2014). A Markov decision process model for coordinating the inventory replenishment and order fulfillment of an e-commerce retailer. European Journal of Operational Research, 239(1), 31-43. [32] Lu, S., & Chan, F. T. (2016). Joint optimization of production and inventory control under lead time uncertainty and demand fluctuation. International Journal of Production Economics, 180, 49-59. [33] Leachman, R. C., Boyer, K. K., & Vandenbosch, M. B. (1991). Supply chain inventory control: A comparison among JIT delivery, EOQ and the EOQ with production smoothing. International Journal of Production Research, 29(9), 1887-1905. [34] Shih, H. S., & Hsu, C. Y. (2007). A supply chain inventory control model with demand and supply uncertainties. International Journal of Production Economics, 108(1-2), 2-13. [35] Moinzadeh, K., & Moghaddam, M. (2012). Coordinating a three-level supply chain with joint replenishment and Markovian deterioration under fuzzy random environment. Applied Mathematical Modelling, 36(7), 3036-3046. [36] Giri, B. C., Bardhan, S., & Maiti, M. (2017). Optimization of fuzzy random EPQ inventory model with lead time using particle swarm optimization. Applied Soft Computing, 50, 130-142. [37] Soni, H., Jha, P. C., & Singh, S. (2016). Joint optimization of production, maintenance, and inventory control policies for a manufacturing system under two-level trade credit financing. International Journal of Production Research, 54(3), 711-730. [38] Amin, R., Anwer, M. M., & Zaman, U. (2019). Vendor managed inventory system for a multi-item deteriorating supply chain with stochastic demand. International Journal of Production Research, 57(10), 3285-3310. [39] Kapoor, S., Singh, P., & Verma, V. (2018). An optimal ordering policy for a multi-item, multi-supplier, and multi- retailer supply chain system under price dependent demand. International Journal of Production Research, 56(9), 3417-3438. [40] Leng, M., Yang, X., & Xie, J. (2017). An inventory management model with dynamic order crossover in e-commerce supply chains. Journal of the Operational Research Society, 68(7), 788-801. [41] Nof, S. Y., & Sharma, S. (2013). Inventory management in reverse logistics under uncertainty: A Markov decision process approach. Journal of Intelligent Manufacturing, 24(1), 45-62. [42] Phan, C. D., Huynh, V. N., & Nguyen, N. (2016). Two-warehouse inventory control system with Markovian demand and limited order fulfillment rate. European Journal of Operational Research, 249(3), 940-952. [43] Baboli, A., & Biazaran, M. (2015). A novel fuzzy goal programming approach for multi-product multi-echelon supply chain master planning problem with multiple objectives. Expert Systems with Applications, 42(4), 1990-2003. [44] Baradaran, F., & Seyedhoseini, S. M. (2017). Optimizing single- and multi-item vendor managed inventory systems with stochastic demand using simulated annealing algorithm. Computers & Industrial Engineering, 105, 70-87. [45] Gao, L., & Ma, X. (2019). An inventory control problem with lead-time-dependent demand and emergency order. International Journal of Production Economics, 216, 273-282. [46] Günalay, Y., & Yücel, A. (2019). Stochastic dynamic programming models for perishable inventory systems under Markovian demand processes. International Journal of Production Economics, 213, 119-131. [47] Paterson, C., & Soh, C. (2008). A Markov decision process model for managing product returns. Computers & Operations Research, 35(4), 1080-1094. [48] Shah, N. H., Chen, Y., & Petruzzi, N. C. (2018). Optimizing inventory control decisions under dynamic demand learning. Management Science, 64(2), 894-914. [49] Si, C., & Zhu, M. (2018). Analysis of coordinated policies for perishable inventory systems with Markovian demand processes. European Journal of Operational Research, 271(3), 1083-1093. [50] Xie, J., Leng, M., Tang, Y., & Qi, Y. (2018). Inventory control with emergency replenishment for a retailer under combined promotion and low fill rate. International Journal of Production Economics, 198, 126-139. [51] Zhang, J., & Chen, J. (2016). Optimal production and inventory control policies for a deteriorating item with a finite production rate. Omega, 58, 95-108. [52] Zou, C., Xu, L., & Yue, X. (2020). Optimal inventory control policy for a perishable product with multiple demand classes. European Journal of Operational Research, 287(3), 1156-1169. [53] Zhang, H., Li, N., & Lin, J. (2024). Modeling the Decision and Coordination Mechanism of Power Battery Closed- Loop Supply Chain Using Markov Decision Processes. Sustainability, 16(11), 4329. https://doi.org/10.3390/su16114329 [54] Dehghanian, F., & Svensson, G. (2017). Optimal inventory control of a single item with continuous-review replenishment policy under nonstationary stochastic demand. International Journal of Production Economics, 191, 111-124. [55] Huang, Z., & Liang, L. (2018). A two-period inventory control model under risk aversion and delayed payment. Computers & Industrial Engineering, 115, 511-518. 7864 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 7846-7864, 2024 DOI: 10.55214/25768484.v8i6.3714 © 2024 by the author; licensee Learning Gate [56] Mahmoodi, M., & Baradaran, F. (2016). An integrated supply chain inventory model with stochastic demand and carbon tax. Transportation Research Part E: Logistics and Transportation Review, 90, 146-164. [57] Nascimento, A. R., Ferreira, L. L., Ferreira, L. A. F., & Ferreira, D. C. (2018). Optimal inventory policy for perishable products with time-varying demand and costs. European Journal of Operational Research, 264(3), 1009-1019. [58] Petruzzi, N. C., & Dada, M. (1999). Pricing and the newsvendor problem: A review with extensions. Operations Research, 47(2), 183-194. [59] Rahmaniani, R., Moinzadeh, K., & Ahmadi, M. (2020). Optimal decision-making for single-product multi-echelon supply chain considering strategic customer behavior. Journal of Industrial Engineering and Management Studies, 7(2), 62-77. [60] Sucky, E., & Sucky, E. (2016). Supply chain optimization under inventory control in the automotive industry. International Journal of Production Economics, 171(Part 3), 339-347. [61] Vakharia, A. J., & Trivedi, V. (1999). Optimal pricing and trade credit terms. Journal of Business, 72(2), 183-201.