248 American Academic Scientific Research Journal for Engineering, Technology, and Sciences ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 https://asrjetsjournal.org/index.php/American_Scientific_Journal/index Modern Trends in Automating ETL Pipelines in Azure Sree Hari Subhash * Senior Data Engineer, Dallas, USA Email: sreemahn@outlook.com Abstract The study is aimed at systematizing and analyzing contemporary trends in the automation of ETL pipelines within the Microsoft Azure cloud ecosystem. The objective of the work is to identify key paradigms, toolsets and architectural approaches, as well as to develop a scientifically grounded model for selecting an optimal technology stack depending on the specifics of business tasks. The methodological basis includes a comprehensive analysis of current scientific publications, technical documentation and industry reports, as well as a comparative evaluation of the leading Azure services: Data Factory, Databricks and the newest Microsoft Fabric platform. As a result of the study, the dominant trends have been identified: the shift from classical ETL to ELT, large-scale adoption of serverless architectures, active development of low-code/no-code solutions and the emergence of the Data Lakehouse concept as a universal data repository. Within the framework of the work, a decision matrix for selecting an automation tool is proposed, based on the criteria of transformation complexity and the need for an integrated analytics platform. It is concluded that the evolution of automation tools in Azure is progressing from a set of disparate services toward fully integrated platform solutions, which fundamentally changes the methodology of data lifecycle design and management. The results of the study are of practical value for data architects and engineers, as well as for IT department leaders responsible for developing and implementing data management strategies in a cloud environment. Keywords: ETL; Azure; automation; Microsoft Fabric; Azure Data Factory; Azure Databricks; Data Lakehouse; ELT; data pipeline; serverless. 1. Introduction As a result of the digital transformation of the economy and society, the volumes of data generated have grown to unprecedented scales. According to estimates, by 2025 their aggregate volume will exceed 175 zettabytes, which opens significant opportunities for business and simultaneously creates serious infrastructure challenges [1]. ------------------------------------------------------------------------ Received: 8/25/2025 Accepted: 10/9/2025 Published: 10/20/2025 ------------------------------------------------------------------------ * Corresponding author. https://asrjetsjournal.org/index.php/American_Scientific_Journal/index American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 249 Effective use of such volumes for informed decision-making, personalization of customer experience and optimization of business processes is possible only in the presence of reliable and easily scalable data- integration mechanisms. Historically, the task of data consolidation, cleansing and loading was solved using ETL pipelines. However, traditional on-premise solutions demonstrate limited scalability, high capital and operational expenditures, as well as insufficient processing speed, failing to meet modern requirements for real-time analytics [2]. Migration to cloud platforms such as Microsoft Azure removes many of these limitations by providing elastic, managed and cost-effective services for building and operating data pipelines The relevance of the research is determined by the rapid evolution and diversification of ETL-automation tools within the Azure environment, which complicates the selection of an optimal technology stack for data architects and engineers. The scientific gap lies in the absence of comprehensive comparative evaluations of the latest platforms, in particular Microsoft Fabric, relative to established solutions such as Azure Data Factory and Azure Databricks The aim of this work is to identify key paradigms, tooling and architectural approaches, as well as to develop a scientifically grounded model for selecting the optimal technology stack depending on the specifics of business tasks. The scientific novelty consists in the description of a multi-criteria decision-making model that, unlike existing review papers, compares competing Azure technologies (Data Factory, Databricks, Fabric) according to parameters such as performance, cost, entry threshold and integration capabilities The author’s hypothesis is based on the assumption that the evolution of automation tools in Azure is shifting from isolated ETL utilities towards integrated unified analytics platforms such as Microsoft Fabric, which fundamentally transforms approaches to data-lifecycle design and management A limitation of this study is the deliberate focus exclusively on the Microsoft Azure ecosystem without direct comparison with alternative public clouds, which limits the external comparability of the results. 2. Materials and methods In recent years, there has been a clear shift toward unification and automation of data extraction, transformation and loading (ETL) processes at the enterprise level. Thus, Reinsel, Gantz and Rydning in an IDC report show that volumes of generated and consumed data are growing rapidly, and requirements for end-to-end processing – from peripheral IoT devices to central repositories – are exerting pressure on existing architectures and automation tools [1]. Nambiar A., Mundra D. [2] give a detailed comparative review of classical data warehouses and data lakes, emphasizing that modern solutions increasingly combine the strengths of both approaches, integrating metadata, schemas and security mechanisms into hybrid platforms. In the same vein, Armbrust M. and his colleagues [4] propose the lakehouse concept – an open platform combining the capabilities of a traditional Data Warehouse and advanced analytical frameworks, enabling the deployment of American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 250 automated ETL pipelines on top of a unified storage and compute infrastructure. The conducted studies make a fundamental contribution to the conceptualization of architecture evolution trajectories; however, their generalizing conclusions require clarification with respect to production scenarios in Azure. In particular, IDC forecasts [1] operate at macro-level trends of data growth and do not provide representative metrics on integration and observability overheads in specific cloud stacks. Source [2] correctly captures the convergence of DW and DL approaches, but does so primarily at the level of logical models, with little attention to issues of end-to-end lineage and schema governance under frequent releases. The lakehouse architecture as formulated in source [4] provides a methodological framework, yet leaves open practical aspects of multi-team operation (ownership models for the silver/gold zone, delineation of responsibilities across SRE/DE/BI), which motivates the emphasis on operational criteria for tooling selection in Azure. Directly within the Azure ecosystem the main works focus on the development and interaction of orchestration, integration and analytic services. Tirupati K. K. and his colleagues [6] systematically describe practices for building data pipelines using Azure Logic Apps and Azure Data Factory (ADF): the authors demonstrate a modular approach to workflow construction with the ability to seamlessly connect resource groups, version controls and alerts, which contributes to more flexible automation and rapid debugging of pipelines. Singu S. K. [7] considers the integration of Azure Data Factory with the Databricks platform: the study shows how containerized Spark tasks within Databricks can implement horizontally scalable ETL processes that automatically adjust to workload and data volumes while providing a unified logging and monitoring mechanism. Borra P. [8] highlights the new Microsoft Fabric platform, analyzing its architecture and its ability to combine streaming, batch and interactive analytics within a single management interface; it is noted that Fabric expands on lakehouse ideas but adds built-in automation tools such as preconfigured pipeline templates and deep integration with Power BI for real-time visualization of ETL results. The presence of detailed pipeline descriptions in sources [6–8] creates a useful catalog of patterns, while at the same time revealing a number of limitations of prior work. First, the presented cases — singular and time-dependent: changes in versions of Integration Runtime, Spark runtimes, or security policies can alter the observed performance and cost, which complicates the reproducibility of the results. Second, studies [6] and [7] focus predominantly on the functional coupling of services, bypassing comparative analysis of TCO and elasticity under variable load; publication [8] elucidates the integrity of Fabric, but addresses only declaratively the issues of artifact migration and compatibility with existing ADF pipelines. Several publications examine related methods and approaches capable of enriching automation toolkits. Habib G. and his colleagues [9] explore the potential of integrating blockchain technologies with cloud platforms, including Azure, to ensure data immutability and traceability in ETL pipelines; the focus is on smart contract mechanisms and distributed ledgers as means of automatic change tracking and guaranteeing data quality in intercompany exchanges. Pavao A. and his colleagues [3] demonstrate how open platforms for scientific competitions (Codalab) stimulate the development of data processing algorithms and tools by setting common benchmarks and architectural templates that can be adapted to Azure pipelines to compare the effectiveness of various ETL strategies. Shutaywi M., Kachouie N. N. [5] propose using silhouette analysis to assess clustering quality in data transformation processes, which allows automatic selection of optimal algorithm parameters for partitioning and thus improves the accuracy and consistency of resulting datasets. The aforementioned American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 251 directions expand the automation toolkit, yet require applicability assessment. Blockchain integration described in source [9] increases traceability, but imposes transactional and operational costs (commit latency, key management, compliance with personal data protection requirements) that are justified not for all domains and rarely align with the principle of a minimally sufficient trusted environment within OneLake. The use of competitive platforms described in source [3] is productive for the standardization of datasets and metrics, but transferring benchmarks into the corporate perimeter requires the institutionalization of annotation processes and repeatable pipeline packaging (artifact registry, isolated telemetry). The transformation quality assessment methods based on the silhouette coefficient in source [5] are promising for automatic parameter calibration, although they require adaptation to streaming scenarios and to heterogeneous feature spaces characteristic of enterprise data marts. Taken together, the literature demonstrates two key trends: the evolution of architectures from disparate storage solutions to unified lakehouse platforms and the transition from manual scripts to declarative, modular pipelines with a rich set of built-in monitoring and management tools. At the same time, contradictions persist: some authors [4, 8] see the future in unified platforms governed by a single vendor, while others [2, 9] emphasize the importance of openness and cross-platform compatibility. Issues of security and data governance at the cloud– edge interface, as well as hybrid scenarios employing hyperautomation (low-code/no-code) in large enterprises, remain insufficiently addressed. Furthermore, the application of machine learning methods for dynamic adaptation of ETL configurations in response to changing incoming streams and business requirements has not yet been fully developed. 3. Results and Discussion The conducted analysis allows delineating the key development vectors of data pipeline automation in Azure. The evolution of architectures vividly demonstrates the transition from monolithic, poorly scalable solutions to flexible, component-based and unified systems. Initially, the primary automation tool was Azure Data Factory, positioned as a cloud orchestrator whose task was to coordinate the movement of data between storage repositories and to initiate simple transformations via Copy Data activities or by executing scripts in Azure SQL Database [5, 6]. This architecture, depicted in Fig. 1, ensured efficiency for simple scenarios but faced limitations when executing complex and resource-intensive code transformations. American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 252 Figure 1: Simplified architecture of a classic ETL pipeline in Azure using ADF [6] With the rapid proliferation of the Data Lake concept and the growing need to process unstructured and semi- structured data, the functional capabilities of Azure Data Factory (ADF) have ceased to meet modern requirements. This has led to the formation of hybrid architectures in which ADF retains the role of a centralized orchestrator while a distributed engine, Apache Spark within the Azure Databricks service, is employed for heavy computation and transformations. In accordance with the model presented in Figure 2, ADF initiates the pipeline on a schedule or in response to an event trigger, copies raw data from sources into the Data Lake, and delegates control to the corresponding notebook or job in Databricks. The latter performs the primary operations of data cleansing, enrichment, and aggregation, and then loads the resulting output into the target storage— whether a data mart in Synapse Analytics or another DBMS. Effective operation in such a distributed environment requires specialists to possess deep proficiency in PySpark and Spark SQL when working with large datasets. Nevertheless, despite the remarkable power and flexibility of this approach, it intensifies the challenge of integrating and managing a multitude of heterogeneous services. Development teams are compelled to use various interfaces simultaneously, configure secure interaction American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 253 channels between ADF, Databricks, Synapse, Power BI, and Azure Key Vault, and monitor diverse cost models and artifact versioning within each system. Figure 2: Hybrid ELT architecture with Azure Data Factory and Azure Databricks [3, 5, 7] Emerging in 2023, the Microsoft Fabric platform became not merely a new cloud service but a holistic reconceptualization of the principles for constructing an analytical ecosystem in Azure. At its core lies the idea of consolidation — instead of a set of separate, albeit integrated, components, Fabric offers a unified SaaS environment encompassing all key stages of working with data: from ingestion (Data Factory in Fabric) and engineering (Synapse Data Engineering based on Apache Spark) to hybrid storage in the Lakehouse format, event-driven real-time analytics (Synapse Real-Time Analytics) and visualization in Power BI [8, 9] The principal element of this architecture is the global logical store OneLake, common to the entire organization. It provides all of the platform’s compute engines with metadata-level access to data, eliminating the need for physical copying or movement between services. Thanks to this approach, many issues characteristic of earlier generations of hybrid landscapes disappear — from challenges in integration and access- rights reconciliation to data duplication and multi-tier cost accounting. Figure 3 demonstrates how the consolidation of services within Fabric simplifies the data architecture, making it more transparent, manageable and easily scalable. American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 254 Figure 3: Unified Analytics Architecture with Microsoft Fabric [8, 9] For the formalization of the selection among these three main approaches (pure ADF, hybrid ADF+Databricks and unified Fabric) a comparative evaluation was conducted across key criteria, the results of which are compiled in Table 1. The analysis shows that there is no single best solution. The choice represents a compromise and depends heavily on the project context. ADF is ideal for simple orchestration tasks, Databricks is unparalleled in complex computations, and Fabric offers a balance and ease of integration. Based on this analysis, a decision-making model in the form of a matrix can be proposed (Figure 4). This matrix maps two key project factors: the complexity of the required data transformations and the need for a unified analytical environment with tight integration with business intelligence (BI) tools. American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 255 Table 1: Comparative analysis of ETL automation tools in Azure [4, 6, 7, 9] Parameter Azure Data Factory (ADF) Azure Databricks Microsoft Fabric Main scenario Data orchestration, simple ELT operations, low-code development. Large-scale big data processing, complex transformations, ML/AI. Unified analytics from ingestion to BI, collaboration, Lakehouse. Required skills Visual interface, SQL knowledge. Low entry barrier. Programming (Python/PySpark, Scala), Apache Spark, Data Engineering. Mixed: visual interface + Spark/SQL knowledge. Simplified onboarding. Performance Limited for complex transformations (uses Mapping Data Flows based on Spark, but with less flexibility). Highest, fine-tuned Spark clusters for maximum performance. High (uses optimized Spark), but may lag behind a finely tuned Databricks. Cost model Pay-per-activity execution, Integration Runtime hours. Pay-per-cluster runtime (DBU – Databricks Unit). Requires monitoring. Single capacity-based model (Capacity Units). More predictable. Integration Excellent with sources, but requires manual integration with other services (Databricks, Synapse). Native integration with Data Lake, but requires setup for connectivity to other services. Seamless internal integration of all components. Unified interface. AI/ML integration Limited (execution of Azure ML pipelines). Best-in-class. Native MLflow support, integration with Azure AI Studio. Deep expertise in model building with Keras, TensorFlow, scikit-learn Integrated capabilities, including Copilot. Simplifies AI adoption. American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 256 Figure 4: Decision matrix for choosing an ETL automation tool in Azure [4, 6,7] Thus, based on Figure 4, it should be noted that the axis transformation complexity should be operationalized through a set of indicators: the share of stateful operations, sensitivity to the quality of the distributed scheduler, the requirement for reusability of code artifacts, and the intensity of interaction with external libraries. The axis need for a unified analytical environment is advisable to measure not only by the breadth of the stack but also by coordination costs between teams (the number of integration boundary touchpoints, divergence of access policies, duplication of storage layers). Projecting real projects onto this plane shows: at high values of both axes, Fabric minimizes alignment cost, whereas under extreme computational saturation and moderate integration connectedness, the advantage remains with Databricks under ADF orchestration. The performance–cost trade-off in Azure manifests at different points along the elasticity–predictability curve. In ADF, the upper bound of scaling is constrained by the flexibility of Mapping Data Flows and the throughput of Integration Runtime; in the ADF+Databricks bundle, the key lever is job profiling (cluster size, auto-scaling, American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 257 Shuffle caching) and DBU management discipline; in Fabric, predictability is achieved through the capacity model and the uniform telemetry of OneLake, however for narrowly tailored optimization tasks there may be a lag behind manually tuned Databricks clusters. Practically, this means the choice should be made by minimizing total cost of ownership: compute cost + coordination + observability + risk of defects under schema changes (schema drift). Finally, migration scenarios are reasonably formalized as a stepwise transition: 1. identification of hot data paths and their dependencies; 2. alignment of security policies and data lineage; 3. phased consolidation of artifacts into OneLake and unification of observability pipelines; 4. migration of computationally heavy steps with an assessment of the benefit. Such a strategy reduces operational risk and increases the reproducibility and manageability of pipelines under conditions of the continuous evolution of Azure services. 4. Conclusion Within the scope of this work, three successive stages of architectural development were identified: at the first stage- Azure Data Factory serves as the central orchestrator; at the second stage- hybrid solutions are formed in which Azure Databricks is engaged to process complex computational tasks; at the third and current stage- full consolidation of all components within the Microsoft Fabric environment is achieved. The primary factors underpinning this transformation are the aim to reduce the complexity of managing heterogeneous solutions, expand user capabilities through low-code tools and provide business with accelerated integrated access to real- time analytics A limitation of the present work is the rapid pace of evolution of cloud services which can in a short time lead to the emergence of new features or services capable of altering the current balance of capabilities. As directions for further research, it is proposed to conduct quantitative comparisons of performance metrics and total cost of ownership (TCO) when implementing typical ETL scenarios on the Microsoft Fabric platform in comparison with the hybrid ADF + Databricks architecture as well as to evaluate the impact of built-in AI tools such as Copilot on the efficiency of development and maintenance of data pipelines. References [1]. Reinsel, D., Gantz, J., & Rydning, J. (2018). The digitization of the world – From edge to core. IDC. https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage- whitepaper.pdf (date of request: 05.05.2025). [2]. Nambiar, A., & Mundra, D. (2022). An overview of data warehouse and data lake in modern enterprise data management. Big Data and Cognitive Computing, 6(4), 1–24. https://doi.org/10.3390/bdcc6040132 [3]. Pavao, A., Abid, A., Racah, E., Roesch, K., Yosinski, J., & Vartak, M. (2023). Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf https://doi.org/10.3390/bdcc6040132 American Academic Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) - Volume 103, No 1, pp 248-258 258 24(198), 1-6. [4]. Armbrust, M., Das, T., Zhu, S., Sen, R., Saha, A., Xin, R. S., ... & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. Proceedings of CIDR, 8, 1–8. [5]. Shutaywi, M., & Kachouie, N. N. (2021). Silhouette analysis for performance evaluation in machine learning with applications to clustering. Entropy, 23(6), 1–17. https://doi.org/10.3390/e23060759 [6]. Tirupati, K. K., Hussain, M. A., Alam, M. M., Rauf, H. T., & Alshazly, H. (2023). Advanced techniques for data integration and management using Azure Logic Apps and ADF. International Journal of Progressive Research in Engineering Management and Science, 3(12), 460–475. [7]. Singu, S. K. (2021). Designing scalable data engineering pipelines using Azure and Databricks. ESP Journal of Engineering & Technology Advancements, 1(2), 176–187. [8]. Borra, P. (2024). Microsoft Fabric review: Exploring Microsoft's new data analytics platform. International Journal of Computer Science and Information Technology Research, 12(2), 34–39. https://doi.org/10.5281/zenodo.11502585 [9]. Habib, G., Hussain, M., Faheem, M., Rauf, H. T., Alshazly, H., & Alharbi, A. (2022). Blockchain technology: Benefits, challenges, applications, and integration of blockchain technology with cloud computing. Future Internet, 14(11), 1–22. https://doi.org/10.3390/fi14110341 https://doi.org/10.3390/e23060759 https://doi.org/10.5281/zenodo.11502585 https://doi.org/10.3390/fi14110341