Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 9, No. 1, 2024 46 Machine Learning Through Physics–Informed Neural Networks: Progress and Challenges Klapa Antonion1, Xiao Wang1, *, Maziar Raissi2, Laurn Joshie2 1Department of Technology of Computers and Communications, University of Extremadura, 10071 Caceres, Spain 2Department of Applied Mathematics, The University of Sydney, Sydney, NSW 2006, Australia *Corresponding author: Xiao Wang, xiao.wang@unex.es Abstract: Physics-Informed Neural Networks (PINNs) represent a groundbreaking approach wherein neural networks (NNs) integrate model equations, such as Partial Differential Equations (PDEs), within their architecture. This innovation has become instrumental in solving diverse problem sets including PDEs, fractional equations, integral-differential equations, and stochastic PDEs. It's a versatile multi-task learning framework that tasks NNs with fitting observed data while simultaneously minimizing PDE residuals. This paper delves into the landscape of PINNs, aiming to delineate their inherent strengths and weaknesses. Beyond exploring the fundamental characteristics of these networks, this review endeavors to encompass a wider spectrum of collocation-based physics-informed neural networks, extending beyond the core PINN model. Variants like physics-constrained neural networks (PCNN), variational hp-VPINN, and conservative PINN (CPINN) constitute pivotal aspects of this exploration. The study accentuates a predominant focus in research on tailoring PINNs through diverse strategies: adapting activation functions, refining gradient optimization techniques, innovating neural network structures, and enhancing loss function architectures. Despite the extensive applicability demonstrated by PINNs, surpassing classical numerical methods like Finite Element Method (FEM) in certain contexts, the review highlights ongoing opportunities for advancement. Notably, there are persisting theoretical challenges that demand resolution, ensuring the continued evolution and refinement of this revolutionary approach. Keywords: Physics–Informed Neural Networks, Machine Learning, Numerical Methods, Partial Differential Equations. 1. Introduction Deep neural networks have excelled across diverse domains like remote sensing[1, 2], computer vision[3, 4], risk prevention[5], optimization[6, 7], pattern recognition[3, 8, 9], regression tasks[10] and Hyperspectral Imaging[11-14] are executed. In applied mathematics, the application of deep learning techniques to classical problems, such as partial differential equations (PDEs), marks a significant trend. Traditional numerical approaches become difficult when faced with PDEs featuring substantial nonlinearities, convection dominance, or shocks. Enter deep learning, presenting a new paradigm in scientific computing due to the remarkable universal approximation and expressiveness of neural networks. Recent studies underscore the potential of deep learning in constructing meta-models for rapid predictions in dynamic systems, adeptly capturing intricate nonlinear input-output relationships. Yet, grappling with the intricacies of high-dimensional complex systems encounters the challenge of dimensionality, a hurdle notably articulated by Bellman in optimal control problems. Nevertheless, machine learning-based algorithms offer promising prospects for tackling PDEs [15]. Forecasts suggest that machine learning-driven PDE- solving methods will remain a focal point as deep learning evolves through methodological, theoretical, and algorithmic advancements[15, 16]. Early attempts at solving differential equations employed simplistic neural network models like Multi-Layer Perceptron’s (MLPs) with minimal hidden layers[17]. Contemporary approaches, leveraging NN techniques, harness optimization frameworks and auto- differentiation techniques, exemplified by Berg and Nyström's unified deep neural network method for estimating PDE solutions[16, 18, 19]. Additionally, the envisioned potential of deep neural networks to construct interpretable hybrid Earth system models for Earth and climate sciences further underscores their significance[18]. Presently, the literature lacks a standardized nomenclature for integrating prior physical knowledge with deep learning. Terms like 'physics-informed,' 'physics-based,' 'physics- guided,' and 'theory-guided' are variably used. Some researchers[20] introduced a comprehensive taxonomy, termed 'informed deep learning,' organizing it into three core conceptual stages: delineating the type of deep neural network used, representing physical knowledge, and integrating this information. Inspired by their framework, this exploration will delve into Physics-Informed Neural Networks (PINNs), introduced in 2017, elucidating how neural network features are employed, how physical information is incorporated, and which physical problems have found resolution within the literature. 2. Physics-Informed Neural Networks Concept Physics-Informed Neural Networks (PINNs) serve as a potent scientific machine learning tool tailored for solving problems entrenched in Partial Differential Equations (PDEs). The crux of PINNs lies in approximating PDE solutions by training neural networks to minimize a loss function that encapsulates critical elements: initial and boundary conditions across the space-time domain, alongside the PDE residual pinpointed at selected collocation points within the domain. These deep-learning networks, constituting PINNs, yield estimated solutions at designated points within a differential equation's integration domain post-training. 47 What sets PINNs apart is the incorporation of a residual network encoding the governing physics equations, a hallmark novelty. The training process of PINNs functions as an unsupervised strategy, eschewing the need for labeled data stemming from prior simulations or experiments. Essentially, the PINN algorithm operates as a mesh-free technique, transforming the direct resolution of governing equations into an optimization problem concerning a loss function. This method intricately integrates the mathematical model into the neural network and reinforces the loss function by introducing a residual term derived from the governing equation. This term acts as a constraining element, narrowing down the acceptable solution space. Unlike approaches reliant solely on data-driven solutions, PINNs prioritize the inherent PDE physics over mere data fitting via neural networks. This concept traces its roots to earlier research [21], shedding light on the potential of leveraging structured prior knowledge. Some researcher[22] leveraged Gaussian process regression to construct representations of linear operator functionals, effectively inferring solutions and providing uncertainty estimates for diverse physical problems—a concept later extended in [22]. The inception of PINNs in 2017 marked a milestone—a new class of data-driven solvers introduced through a comprehensive two-part article, subsequently amalgamated in 2019 [23], which elucidate the PINN approach's prowess in solving nonlinear PDEs like Schrödinger, Burgers, and convection-diffusion equations[24-28]. Their innovation with physics-informed neural networks (PINNs) extends beyond forward problems, encompassing inverse problems where model parameters are learned from observable data. The integration of prior knowledge into machine learning algorithms isn't entirely novel. Early pioneers like Dissanayake and Phan-Thien [29] could be considered among the initial forerunners of PINNs. Building upon the universal approximation advancements of the late 1980s, methodologies emerged in the early 90s proposing neural network approximations for PDEs, such as constrained neural networks[30]. This early network comprised two hidden layers with 3, 5, or 10 nodes per layer, utilizing point- collocation to approximate the L2 error on the domain's interior and boundary. Evaluating the loss function employed a quasi-Newtonian approach, while gradients were assessed using finite differences. 3. Components of PINN Physics-informed neural networks (PINNs) offer a potent solution for tackling problems characterized by limited or noisy experimental data. These networks possess the unique capability to incorporate known data while adhering rigorously to specified physical laws encoded by complex nonlinear partial differential equations, thereby operating as a versatile tool within the realm of supervised learning[31]. In their essence, PINNs excel at solving differential equations across a broad spectrum of formulations, presenting a robust approach to various problem types, like: 𝐹 𝑢 𝑧 ; 𝛾 𝑓 𝑧 𝑧 𝑖𝑛 Ω, 𝐵 𝑢 𝑧 𝑔 𝑧 𝑧 𝑖𝑛 𝜕Ω defined on the domain Ω ⊂ Rd with the boundary 𝜕Ω. In the spatial-temporal coordinate vector z: = [x1..., xd−1; t], u symbolizes the elusive solution, γ stands for the parameters entwined with the physics, f delineates the problem's data, and F embodies the nonlinear differential operator. Moreover, considering the initial condition as a form of Dirichlet boundary condition within the spatial-temporal domain allows denoting B as the operator signifying arbitrary initial or boundary conditions associated with the problem, while g represents the boundary function. These boundary conditions may encompass Dirichlet, Neumann, Robin, or periodic boundary conditions. Above Equation embraces the capacity to depict a myriad of physical systems, encompassing both forward and inverse problems. In the realm of forward problems, the objective is to ascertain the function u for each z, with specified parameters γ. However, in the inverse problem, determining γ from the available data becomes imperative. A comprehensive operator-based mathematical formulation of Eq. (1) can be explored in Mishra and Molinaro's work [32]. Within the PINN framework, the computational prediction of u(z) transpires through a neural network (NN) parametrized by a set of parameters θ, giving rise to an approximation uˆθ (z) ≈ u(z), where (ˆ·) θ signifies an NN approximation realized through θ. In this comprehensive context where both forward and inverse problems find analysis within the same framework, and recognizing PINN's efficacy in tackling both, θ assumes the role of representing the vector encompassing all unknown parameters within the neural network, constituting the surrogate model, and concurrently encapsulating unknown parameters γ in the inverse problem scenario. Within this framework, the neural network must assimilate the approximation of differential equations by discerning θ, which defines the neural network, accomplished by minimizing a weighted loss function contingent upon the differential equation LF, the boundary conditions LB, and potentially known data L, each suitably weighted and considered: 𝜃 ∗ 𝑎𝑟𝑔 𝑚𝑖𝑛 𝜔𝐹 𝐿𝐹 𝜃 𝜔𝐵𝐿𝐵 𝜃 𝜔𝑑 𝐿 𝜃 . In essence, PINN embodies an unsupervised learning methodology when trained exclusively on physical equations and boundary conditions to address forward problems. However, in scenarios involving inverse problems or when certain physical properties are inferred from potentially noisy data, PINN can seamlessly transition into a supervised learning approach. 4. Differential Equations The inaugural vanilla PINN [23] was engineered to solve intricate nonlinear PDEs formulated as ut + Fx u = 0, where x represents a spatial coordinate vector, t signifies the time coordinate vector, and Fx denotes a nonlinear differential operator concerning spatial coordinates. Initially, the PINN architecture demonstrated its proficiency in handling both forward and inverse problems. Over the subsequent years, PINNs have expanded their application scope, venturing into solving a diverse array of equations encompassing ordinary differential equations (ODEs), partial differential equations (PDEs), Fractional PDEs, integro-differential equations (IDEs), and stochastic differential equations (SDEs). This section endeavors to spotlight the trajectory of research advancements in addressing various equation types, classifying them based on their structures and outlining seminal contributions from the literature that harnessed PINN for such problem domains. The exploration commences with PINN studies tackling ODEs, progressing then to investigations on steady-state PDEs including Elliptic type equations, steady-state diffusion, and the Eikonal equation. Subsequently, the examination delves into the domain of the 48 Navier–Stokes equations, followed by an array of dynamic problems such as heat transport, advection-diffusion-reaction systems, hyperbolic equations, and Euler equations or quantum harmonic oscillator. Concluding this overview, the section delves into Bayesian problems associated with the previously addressed PDEs, shedding light on strategies to navigate uncertainties inherent in stochastic equations. 5. Application and Perspective The preceding sections delved into the neural network aspect of the PINN framework and highlighted the range of equations addressed in existing literature. This section initiates by exploring the management of physical information within the PINN framework—an examination into how data and models intertwine to enhance efficacy. Subsequently, it scrutinizes real-world applications of PINNs, shedding light on software platforms like DeepXDE, NeuralPDE, NeuroDiffEq, and others, all of which emerged in 2019, fostering advancements in PINN design. The future trajectory of PINN's theoretical or applied configurations remains uncertain. Here, we can evaluate incomplete findings from papers, scrutinize the most contentious aspects of PINNs, identify unexplored realms, and delineate intersections with other disciplines. Despite significant strides in augmenting PINN capabilities through published works, numerous unresolved issues persist. These encompass a broad spectrum, spanning from theoretical considerations—such as convergence and stability—to implementation challenges, including boundary condition management, neural network design, general PINN architecture, and optimization aspects. While PINNs and other deep learning methods leveraging physics priors hold promise in solving high-dimensional PDEs prevalent in physics, engineering, and finance, they encounter hurdles in accurately approximating solutions compared to specialized numerical methods tailored for specific PDEs. Particularly, PINNs may struggle with learning intricate physical phenomena, such as solutions exhibiting multi-scale, chaotic, or turbulent behaviors. 6. Conclusion This review serves as an in-depth exploration of the innovation process in the field of PINNs over the past four years, transcending the boundaries of a mere research survey. Raissi's seminal research [143, 144], pioneering the PINN framework, initially focused on employing PINNs to solve established physical models. These groundbreaking papers propelled the PINN methodology into the spotlight, further substantiating its original conceptual framework. Across the analyzed studies, efforts were concentrated on tailoring PINNs through adjustments in activation functions, gradient optimization techniques, neural network architectures, and the structures of loss functions. An intriguing extension of the original PINN concept involved utilizing minimal model information within the physical loss function, bypassing typical PDE equations, while concurrently embedding the validity of initial or boundary conditions directly into the NN structure. Fewer studies delved into alternatives to automatic differentiation or grappled with convergence issues[33]. A pivotal subset of publications aimed to elevate this progression by introducing comprehensive frameworks catering to various sub-types of physical problems or multi-physics systems[33]. The foundational brilliance of the initial PINN articles lies in reviving the optimization of problems with physical constraints, first approximating the unknown function using a neural network, and subsequently expanding this approach into a hybrid data-equation-driven methodology within contemporary research. Prior studies in past years employed diverse methods to approximate the unknown function, such as kernel approaches or utilizing PDE functions as constraints in optimization problems[34]. Nonetheless, PINNs are inherently grounded in physical information, derived either from data point values—usually confined to initial or boundary data—or from collocation points enforcing compliance with the physical model equation. This survey delves into the evolution of PINNs from their inception in the pioneering works and tracks the progression of integrating physical priors into Neural Networks. It scrutinizes PINNs as a collocation-based method for solving differential equations using Neural Networks, encompassing various iterations such as the variational PINN (VPINN) in its soft and hard form, incorporating initial and boundary conditions into loss functions or embedding boundary conditions within the neural network structure. The survey meticulously dissects the PINN pipeline, examining each fundamental component: neural networks, the construction of loss functions based on physical models, and the feedback mechanism. It provides an overarching analysis of examples where the PINN methodology has been applied and offers insights into concrete applications of PINNs along with available software packages. However, despite significant advancements, numerous opportunities for improvement persist, especially in unresolved theoretical quandaries. There remains untapped potential for optimizing PINN training and extending their capabilities to tackle multiple equations effectively. References [1] J. Wu, N. Hovakimyan, J. Hobbs, Genco: An auxiliary generator from contrastive learning for enhanced few-shot learning in remote sensing, arXiv preprint arXiv:2307.14612 (2023). [2] Y. Tang, J. Yan, Y. Wu, J. Hong, L. Xu, Z. Lin, Design of Remote Sensing Image Data Analysis and Processing Platform Based on Environmental Monitoring, Journal of Physics: Conference Series, IOP Publishing, 2021, p. 012056. [3] J. Wu, D. Pichler, D. Marley, D. Wilson, N. Hovakimyan, J. Hobbs, Extended Agriculture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysis, arXiv preprint arXiv:2303.02460 (2023). [4] J. Wu, J. Hobbs, N. Hovakimyan, Hallucination improves the performance of unsupervised visual representation learning, Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 16132-16143. [5] Z. Xu, Analysis and research on financial risk prevention based on artificial intelligence algorithms, Proceedings of the 2nd International Conference on Bigdata Blockchain and Economy Management, ICBBEM 2023, May 19–21, 2023, Hangzhou, China, 2023. [6] R. Tao, P. Zhao, J. Wu, N.F. Martin, M.T. Harrison, C. Ferreira, Z. Kalantari, N. Hovakimyan, Optimizing crop management with reinforcement learning and imitation learning, arXiv preprint arXiv:2209.09991 (2022). [7] J. Huo, Y. Wang, N. Wang, W. Gao, J. Zhou, Y. Cao, Data- driven design and optimization of ultra-tunable acoustic 49 metamaterials, Smart Materials and Structures 32(5) (2023) 05LT01. [8] Y. Wang, J. Wu, N. Hovakimyan, R. Sun, Double Dynamic Sparse Training for GANs, arXiv preprint arXiv:2302.14670 (2023). [9] Y. Tang, K. Yoshihiro, A. Tohru, B. Danushka, BS-1-12 Analyzing Patterns from Twitter Happiness referring Stock Market and American Unemployment Rate Announcement, Proceedings of the Electronic Information and Communication Society Comprehensive System Conference 2013(2) (2013) " S-23"-" S-24". [10] J. Wu, R. Tao, P. Zhao, N.F. Martin, N. Hovakimyan, Optimizing nitrogen management with deep reinforcement learning and crop simulations, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 1712-1720. [11] Y. Tang, S. Song, S. Gui, W. Chao, C. Cheng, R. Qin, Active and Low-Cost Hyperspectral Imaging for the Spectral Analysis of a Low-Light Environment, Sensors 23(3) (2023) 1437. [12] D. Huang, Y. Tang, R. Qin, An evaluation of PlanetScope images for 3D reconstruction and change detection– experimental validations with case studies, GIScience & Remote Sensing 59(1) (2022) 744-761. [13] S. Gui, R. Qin, Y. Tang, Sat2lod2: A Software For Automated Lod-2 Modeling From Satellite-Derived Orthophoto And Digital Surface Model, arXiv preprint arXiv:2204.04139 (2022). [14] X. Lai, Y. Tang, L. Li, P. Chan, Q. Zeng, Study on microclimate observation network for urban unit: A case study in a campus of Shenzhen, China, Physics and Chemistry of the Earth, Parts A/B/C 110 (2019) 117-124. [15] J. Blechschmidt, O.G. Ernst, Three ways to solve partial differential equations with neural networks—A review, GAMM‐Mitteilungen 44(2) (2021) e202100006. [16] J. Huo, Y. Wang, Y. Cao, 3D computational study of arc splitting during power interruption: the influence of metal vapor enhanced radiation on arc dynamics, Journal of Physics D: Applied Physics 54(8) (2020) 085502. [17] I.E. Lagaris, A. Likas, D.I. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations, IEEE transactions on neural networks 9(5) (1998) 987-1000. [18] J. Berg, K. Nyström, A unified deep artificial neural network approach to partial differential equations in complex geometries, Neurocomputing 317 (2018) 28-41. [19] J. Huo, Y. Cao, Computational Study of the Arc Splitting in Power Interruption: The Effect of the Metallic Vapor on Arc Dynamics, 2020 IEEE Conference on Electrical Insulation and Dielectric Phenomena (CEIDP), IEEE, 2020, pp. 415-418. [20] S.W. Kim, I. Kim, J. Lee, S. Lee, Knowledge Integration into deep learning in dynamical systems: an overview and taxonomy, Journal of Mechanical Science and Technology 35 (2021) 1331-1342. [21] H. Owhadi, Bayesian numerical homogenization, Multiscale Modeling & Simulation 13(3) (2015) 812-828. [22] M. Raissi, G.E. Karniadakis, Hidden physics models: Machine learning of nonlinear partial differential equations, Journal of Computational Physics 357 (2018) 125-141. [23] M. Raissi, P. Perdikaris, G.E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational physics 378 (2019) 686-707. [24] J. Huo, J. Ronzello, A. Rontey, Y. Wang, L. Jacobs, T. Sommerer, Y. Cao, Development of an arc root model for studying the electrode vaporization and its influence on arc dynamics, AIP Advances 10(8) (2020). [25] J. Huo, S. Selezneva, L. Jacobs, Y. Cao, Study of wall ablation on low-voltage arc interruption: The effect of Stefan flow, Journal of Applied Physics 125(21) (2019). [26] J. Huo, S. Selezneva, L. Jacobs, Y. Cao, Low-voltage arc interruption computation: The effect of Stefan flow, 2018 IEEE Conference on Electrical Insulation and Dielectric Phenomena (CEIDP), IEEE, 2018, pp. 318-321. [27] J. Huo, A. Rontey, Y. Wang, L. Jacobs, Q. Chen, N. Wang, S. Ma, Y. Cao, Arc hopping dynamics induced by interfacial negative differential resistance, PNAS nexus 1(3) (2022) pgac129. [28] J. Huo, Y. Cao, Interfacial potential barrier induced constriction and stepwise transition of a dynamic arc root, 2019 IEEE Conference on Electrical Insulation and Dielectric Phenomena (CEIDP), IEEE, 2019, pp. 564-567. [29] M. Dissanayake, N. Phan‐Thien, Neural‐network‐based approximations for solving partial differential equations, communications in Numerical Methods in Engineering 10(3) (1994) 195-201. [30] H. Lee, I.S. Kang, Neural algorithm for solving differential equations, Journal of Computational Physics 91(1) (1990) 110- 131. [31] S. Goswami, C. Anitescu, S. Chakraborty, T. Rabczuk, Transfer learning enhanced physics informed neural network for phase-field modeling of fracture, Theoretical and Applied Fracture Mechanics 106 (2020) 102447. [32] S. Mishra, R. Molinaro, Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs, IMA Journal of Numerical Analysis 42(2) (2022) 981-1022. [33] S. Wang, X. Yu, P. Perdikaris, When and why PINNs fail to train: A neural tangent kernel perspective, Journal of Computational Physics 449 (2022) 110768. [34] M. Hinze, R. Pinnau, M. Ulbrich, S. Ulbrich, Optimization with PDE constraints, Springer Science & Business Media2008.