International Journal of Applied Sciences and Smart Technologies International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 1 Solving Inverse Problems Interestingly Rommel R. Real 1,2 1 Mathematical Sciences Institute, College of Science, The Australian National University, Canberra, Australia 2 Department of Mathematics, Physics, and Computer Science, University of the Philippines Mindanao, Mintal, Davao City, the Philippines Corresponding Author: rommel.real@anu.edu.au (Received 28-06-2019; Revised 17-10-2019; Accepted 17-10-2019) Abstract Inverse problems deal with recovering the causes for a desired or given effect. Their presence across sciences and their theoretical study can be traced to a classic example. Moreover, their ill-posedness motivates computational methods to solve them, and we give a very humble introduction to them. In relation we discuss their active research trends. Keywords: divergence measure, ill-posed equation, inverse problem, regularization, noisy data 1 Introduction Inverse problems are everywhere. The simplest example out there would be human vision. From measurements of scattered light that reaches our retinas, our brain constructs a detailed 3-dimensional image of the world around us. But how is this possible with only a limited number of points in our surroundings? Our brain has a way to complete the image by interpolating and extrapolating the data acquired from the identified points. Constructing this image is actually an example of what is called an inverse problem. International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 2 There is no universal definition to inverse problems. One might say that inverse problems are concerned with determining causes for a desired effect [4]. The American mathematician J.B. Keller gave this well-cited definition [12]: “We call two problems inverses of one another if the formulation of each involves all or part of the solution of the other. Often, for historical reasons, one of the two problems has been studied extensively for some time, while the other is newer and not so well understood”. This implies that for an inverse problem to be defined, first there must be an underlying direct problem. We will consider some examples. 2 An Important Example One of the most well-studied inverse problems is known as Calder ́n’s problem. Back in 1980 A. P. Calder ́n published a paper “On an inverse boundary value problem”. It aims to solve this problem: is it possible to determine the electrical conductivity of a medium by making voltage and current measurements at the boundary of the medium. This inverse method is also known as electrical impedance tomography (EIT). Interestingly, Calder ́n had been researching on this problem while working as a petroleum engineer for Argentina’s state oil company back in 1947. He stayed there for a few years before embarking a full-time career in mathematics. His paper became a ground-breaking one: it started a new era of mathematical research on inverse problem. Later on EIT would find itself applied in medical imaging. In fact, a prize for outstanding contribution in the field of inverse problems is named after him (check out Calderon Prize). You may check out Gunther Uhlmann’s paper [16] for a summary of progress in Calder ́n’s problem. Calder ́n’s problem is a good example on how real-world problems motivate mathematical research. In geophysics, one needs to explore the Earth’s subsurface using reflected seismic waves. In ecology, space monitoring is used to assess trends of biological diversity using reconstructed satellite images. In medical imaging, tumors in a part of human body must be accurately detected from the attenuation of x-rays. Mathematically, these inverse problems are described using operator equations whose International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 3 specific properties need to be studied with rigour. In fact, the progress in mathematical theory of inverse problems is independent of the growing applications, though the gap between them remains wide. This gap can be filled by showing more applications where the theoretical results hold, promoting a deeper appreciation of inverse problems. 3 Regularization An inverse problem can be expressed as an operator equation in the form ( ) , (1) for a specific exact data , be a parameter-to-observation mapping between (they can be Hilbert, Banach, or even a general topological spaces), for a given parameter variable . Inverse problems have one common property that makes them really hard to solve: they are ill-posed. In mathematics, a problem is well-posed if the following three are satisfied: the problem has a solution, it has a unique solution, and the solution depends continuously on the data. If one of the three is violated, the problem becomes ill-posed. For instance, the mathematical problem of EIT consists of a highly nonlinear partial differential equation, which is very ill-posed. To address instability, we can solve the original ill-posed problem by solving a sequence of approximating well-posed problems. This technique is called regularization. Applying some numerical scheme (e.g. the ones from optimisation) to an ill-posed problem may lead to different solutions. Regularisation ensures a specific solution with desirable properties is recovered. For instance, suppose we have a noisy data . Instead of solving the ill-posed equation (1), we solve an equivalent minimisation problem. ( ( ) ) (2) for a an admissible set and * + be the divergence measure between the reconstrution ( ) and the noisy data . Since (1) is ill-posed, (2) is also ill-posed, but easier to handle. Now we describe a very common regularization method. In variational regularization, we solve a surrogate of (2), given by International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 4 ( ( ) ) ( ) (3) where * + is the penalty functional and is a positive constant regularization parameter. Now, (3) is stable due to the promotion of the penalty term, and can be solved using optimisation algorithms. An example of (3) common in statistics and machine learning is Tikhonov regularization, or ridge regression, where normally the penalty term favors solutions with smaller norms, while the functional is the least-squares functional. The parameter serves as a trade-off between data fidelity and stability, and must be fine-tuned. If the regularization parameter is chosen via a given parameter choice rule to solve (3), we can obtain the solution with this best possible trade-off. Moreover, finding an appropriate regularization parameter to solve (3) is a very active of research in inverse problems, and usually done by imposing assumptions on the ill-posed problem. Alternatively, iterative regularization can be used. To describe, let us assume and to be Hilbert spaces with norm given by ‖ ‖. The simplest of this type is Landweber method, which is simply the gradient method applied to solving ‖ ( ) ‖ Starting an initial guess , the (nonlinear) Landweber iteration [5] is given by ( ) ( ( ) ) (4) where is a relaxation constant chosen often between and (can be constant for all ). Since the original problem (1) is ill-posed, (4) must be terminated after a finite number of iterations to ensure the best reconstruction possible. In this case the iteration acts as the regularization parameter. As in variational regularization, choosing an appropriate using an appropriate parameter choice rule is also an active research area. In some problems, variational regularization in terms of (3) may lead to solving large and sparse systems of equations, which can be computationally expensive. In addition, for every value of taken from an interval (e.g. exponential grid), a full numerical scheme must be implemented to solve to solve. In contrast, iterative regularization have lesser computational burden, but could take a large number of iterations. Due to this International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 5 observation, iterative methods are known to be slow but reliable, which makes them more preferred. However, variational regularization has one important advantage: it can be formulated for problems using generalized divergence/residual terms ( ( ) ) where in (3) does not have to be a norm in Banach spaces. Some instances are in EIT, where the measurement error of the data could depends on the location of the measurement (i.e. error is larger in areas closer to the ultrasound receiver). This is also the case when measuring probability measures. For a convenient overview of these methods, the reader may refer to [9,13]. Lately, in some publications theoretical results often in this research area go hand-in- hand with numerical examples. Some numerical examples come from anywhere optimal control, plasma physics, economics, etc. Theoretical results usually involve convergence rates, in terms of the noise level ( ), they allow comparisons between regularization methods. However this is not be doable in some settings, such as for iterative regularization methods in Banach spaces [8,11,14]. The more important type of theoretical result is the proof of regularization property: given a parameter choice rule, the approximate solution produced by the regularization method should converge to the (unknown) desired exact solution as the noise level decays. 4 Current Trends Now i want to cite some important research trends in the field of regularization for inverse problems. One of the most well-studied types of parameter choice rules is the discrepancy principle, which states to pick the regularization parameter or once the residual becomes smaller than a multiple of the noise level. Mathematically, for iterative methods it means to pick such that ( ( ) ) ( ( ) ) for a given . For both variational and iterative regularization methods this has been well-studied [4,5,14]. However, notice that aside from the noisy data , the discrepancy principle requires information of the noise level . In some instances such International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 6 information may not available, and either overestimation or underestimation of the noise level may lead to an unsatisfactory solution of the inverse problem. Hence, for regularization methods, it is necessary to develop heuristic rules which do not need the noise level. Although Bakushinskii’s veto [1] states that heuristic rules cannot lead to convergence in the worst case scenario for any regularization method, it is still possible to prove some convergence results under such rules under specific assumptions on the noisy data. A well-studied heuristic rule was formulated by Hanke and Raus [6] and it has been developed since then, for instance in [7]. Another growing trend is studying regularization methods in Banach spaces. The standard reference book [4] compiles fundamental results under Hilbert spaces. The need for extending these results towards Banach spaces stems out naturally when the sought solution is non-smooth, e.g. when it is sparse or piecewise constant. Hilbert spaces are too smooth to cover such solutions. A strategy is to introduce a nonsmooth penalty term, which includes the -norm and the total variation (TV), in iterative regularization [2,8]. For variational regularization, one way to obtain convergence results is to impose more general conditions on the sought solution [7]. Obtaining results under Banach spaces also allows to cover both linear and nonlinear problems, leading to more computationally challenging examples [15]. This is a great development since many inverse problems are expressed in terms of highly nonlinear partial differential equations, leading to nonlinear and often non-differentiable forward operators. In this area obtaining convergence results using more general assumptions and extending them towards general topological spaces [13] is still very active. Another important issue involves the forward operator. For instance, problem (4) is formulated assuming is differentiable, in a more general sense. It is possible to obtain a modified Landweber method where instead of the derivative of , an operator approximating the derivative can be used, leading to a derivative-free iterative method [10], Chapter [4]. So far only a few types of derivative-approximating operators have been known [3], and different types may lead to different convergence analyses. Exploring more problems with nonsmooth forward operators is always a welcome research area. In some problems where the direct problem involves not fully linear International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 7 PDEs, the corresponding forward operator may be too complicated to formulate, so solving inverse problems with it would be necessary. 5 Conclusion Inverse problems are still starting to gain better understanding, dating back around thirty years. It’s an area that requires expertise in computational maths, such as in solving PDEs to optimisation. Obtaining theoretical result involves lots of estimation in various settings, from Banach spaces to Sobolev spaces! It’s a relatively new area that needs more people to move in! References [1] A. B. Bakushinskii, “Remarks on choosing a regularization parameter using the quasioptimality and ratio criterion,” USSR Computational Mathematics and Mathematical Physics, 24 (4), 181 182, 1984. [2] R. Bot and T. Hein, “Iterative regularization with a general penalty term-theory and applications to and tv regularization,” Inverse Problems, 28 (10), Article ID 104010, 1 19, 2012. [3] C. Clason and V. H. Nhu, “Bouligand-Landweber iteration for a non-smooth ill- posed problem,” Numerische Mathematik, 142 (4), 789 832. [4] H. W. Engl, M. Hanke, and A. Neubauer, Regularization of Inverse Problems, Kluwer Academic Publishers, London, 1996. [5] M. Hanke, A. Neubauer, and O. Scherzer, “A convergence analysis of the landweber iteration for nonlinear ill-posed problems,” Numerische Mathematic, 72 (1), 21 37, 1995. [6] M. Hanke and T. Raus, “A general heuristic for choosing the regularization parameter in ill-posed problems,” Society for Industrial and Applied Mathematics Journal on Scientific Computing, 17 (4), 956 972, 1996. [7] Q. Jin, “Hanke-Raus heuristic rule for variational regularization in banach spaces,” Inverse Problems, 32 (8), Article ID 085008, 1 18, 2016. International Journal of Applied Sciences and Smart Technologies Volume 2, Issue 1, pages 1–8 p-ISSN 2655-8564, e-ISSN 2685-9432 8 [8] Q. Jin, “Landweber-Kaczmarz method in banach spaces with inexact inner solvers,” Inverse Problems, 32 (10), Article ID 104005, 1 26, 2016. [9] B. Kaltenbacher and A. Klassen, “On convergence and convergence rates for ivanov and morozov regularization and application to some parameter identification problems in elliptic PDEs,” Inverse Problems, 34 (5), article ID 055088, 1 24, 2018. [10] B. Kaltenbacher, A. Neubauer, and O. Scherzer, Iterative Regularization Methods for Nonlinear Ill-Posed Problems, Walter de Gruyter GmbH & Co. KG, Berlin, 2008. [11] B. Kaltenbacher, F. Sch ̈fer, and T. Schuster, “Iterative methods for nonlinear ill- posed problems in banach spaces: convergence and applications to parameter identification problems,” Inverse Problems, 25 (6), article ID 065003, 1 19, 2009. [12] J. B. Keller, “Inverse problems,” The American Mathematical Monthly, 83 (2), 107 118, 1976. [13] C. Poschl, Tikhonov Regularization with General Residual Term, Dissertation, Leopold Franzens Universit ̈t Innsbruck, 2008. [14] F. Schöpfer, A. K. Louis, and T. Schuster, ”Nonlinear iterative methods for linear ill-posed problems in banach spaces,” Inverse Problems, 22 (1), 311 329, 2006. [15] T. Schuster, B. Kaltenbacher, B. Hofmann, and K. S. Kazimierski, Regularization methods in Banach Spaces, Walter de Gruyter GmbH & Co. KG, Berlin, 2012. [16] G. Uhlmann, “30 Years of Calder ́n’s Problem”, S ́minaire Laurent Schwartz-EDP et applications, ́cole Polytechnique, 1 25, 2012-2013.