Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 8, No. 3, 2024 25 Overview of Trust‐region Methods Xinyao Li Beijing Normal University-Hong Kong Baptist University United International College. Zhuhai, Guangdong, 519085, China Abstract: The trust-region technique is an optimization algorithm for solving multidimensional nonlinear optimization complications. It is a class of derivative-based optimization approaches that relies on information about the gradient of the objective function and possibly Hessians. The main idea behind the Trust-region method is to remodel the objective function around the present iteration Solution of the inner subproblem. The confidence region is the region around the present iteration where the model is expected to be accurate. At each iteration, the trust region technique uses a quadratic model to estimate the objective function locally (Yuan, 2019). This model is based on the current iteration and the information about the gradient and Hessian. The confidence region limits the distance the algorithm can take from the current iteration to find the next one. The size of the trust region is adjusted dynamically based on the arrangement between the model and the actual performance values. Keywords: The Trust-region Technique; Performance Values; Quadratic Model. 1. Introduction The trust-domain method aims to find a step direction that will minimize the quadratic model in the trust domain, subject to certain constraints; this sub problem is usually solved by methods such as the conjugate gradient technique or the Newton method, the result of the. The step instruction is then used to update the current iteration, and the process is repeated until the stop criterion is met. The Trust region technique is known for solving nonconvex bad-condition optimization difficulties. It balances the objective task of finding a search area and implementing a local strategy. It has been widely used in machine learning, computer vision, and industrial design optimization (More, 2020). 2. Methodology and Basic Theory of the Trust Region Method. The Trust region technique is a robust optimization algorithm for answering multidimensional nonlinear optimization problems. It is beneficial for problems where the objective function is nonconvex or irregular. This article explores the methodology and basic principles behind the confidence-space approach. 2.1. Methodology The trust-region method follows an iterative method to find the optimal result. The main steps involved are as follows. 1) Quadratic Model Construction At each Iteration, the Trust-region method builds a local quadratic model of the objective function around the current iteration. These models are built based on the current iteration, the gradient, and possibly the Hessian of the objective function. The quadratic model approximates the objective function in the local area (Yuan, 2019). 2) Trust Region Definition: A trust region is around the existing iteration where the quadratic model is expected to be accurate. Overall, the size of the trust region depends on the confidence-zone radius, which is appropriately modified during the optimization process. The trust region acts as a step size constraint and ensures the algorithm does not deviate too far from the current iteration. 3) Sub-problem Formulation: In the trust region, a subproblem is formulated to find the step direction that minimizes the quadratic model under the confidence-domain constraint. This subproblem can be expressed as a problem with enhanced optimization or an unconstrained issue with a penalty word. Different methods can be used to solve the sub-problem, such as the conjugate gradient method or Newton's method. 4) Step Direction Update: Once the step direction is obtained from the result of the sub-problem, it is used to inform the current iteration. The step size is determined by minimizing the objective function in the trust domain and the arrangement between the model and the definite functional values . Another Iterate is performed by taking one step in the specified direction (More, 2020). 5) Iterative Process: They repeat constructing a quadratic model, defining a confidence region, formulating a sub-problem, and iterating until the ending principles are met. Stopping principles can be based on achieving a certain tolerance level, number of iterations, or convergence on other standards. The trust-region approach can be brief as follows; Initialization: Select initial dimensions for the optimization variable. Define the initial trust zone radius. Update Rule: Update the iterate xₖ₊₁ based on the step pₖ: xₖ₊₁ = xₖ + pₖ Solve the Trust-region Sub-problem: Minimize the quadratic model within the trust region: min mₖ(p) = f(xₖ) + f(xₖ)ᵀp + ½pᵀBₖp subject to ||p|| ≤ Δₖ, where Δₖ is the trust region radius Update the Iterate: Compute the step pₖ by solving the trust region subproblem: pₖ = argmin mₖ(p) subject to ||p|| ≤ Δₖ Trust Region Update: Adjust the confidence zone radius Δₖ based on the update’s success: If the actual decrease in the objective function is very close to the predicted decrease, increase Δₖ. Otherwise, decrease Δₖ to decrease the size of the confidence region. 26 Convergence Check: Check convergence based on termination criteria such as achieving the desired accuracy, slight gradient, or small confidence region radius. 2.2. Basic Theory The Trust region method is based on optimization theory and numerical analysis. The following are the main theoretical features of the Trust-region approach. 1) Local Model and Global Convergence: The quadratic model provides a local estimate of the impartial function. The confidence-region method ensures global consistency by introducing a confidence-region constraint, which prevents the algorithm from searching in a finite region around the current iteration. This constraint prevents the algorithm from splitting or converging in local optima (More, 2020). 2) Objective Function Reduction: The Trust region technique aims to reduce the objective function at each iteration. It achieves this by finding a step direction that minimizes the quadratic model of the impartial function in the confidence region. Reducing the objective function is essential for retrieving a step and updating the current iteration. 3) Trust-region Update: The trust region radius is adaptively restructured during optimization built on the model's agreement and actual performance values. If the model accurately represents the project, the confidence space can be expanded to find the required area size. Conversely, if the model deviates significantly from the task, the radius of the confidence zone is reduced to focus on smaller regions. 4) Convergence Analysis: The trust-region approach guarantees convergence under certain circumstances. In convergence analysis, the sequence of iterations generated by the algorithm has been presented to meet to a specific point in the objective function or to a local/global minimizer. This analysis is usually based on an objective function with properties such as Lipsitz continuity, convexity, or strong convexity. 3. Different Versions of the Trust- Region Method Over the years, the Trust-zone approach has been studied and modified, resulting in various interpretations and variations. Here are some definitions of the trust region technique. a) Dogleg Method The dogleg technique is an irregular of the trust-region method that combines the advantages of a very steep descent technique and the Gauss-Newton method. A straightforward method (the direction of maximum slope) and a square method (Gauss-Newton direction) are all used to calculate the step path inside the confidence area. The algorithm identifies those solutions that lie on the boundaries of the confidence region if they lie within the confidence area or cross the boundaries of the confidence area and connecting lines—the Gauss-Newton solution of the current iteration. b) Trust-region Reflective Method The trust region Reflective technique is an extension of the trust region technique for forced optimization problems. This includes constraints using projection or reflective strategies to ensure that the step remains within the realm of possibility. The algorithm updates the iteration to satisfy the constraint by first accepting a step in the confidence domain and then projecting it into the feasible set, if necessary. c) Trust-region Interior Point Methods Trust Region Interior Point approaches combine interior point techniques and trust domain approaches. These methods solve constrained optimization problems by converting the unconstrained problem into a sequence using constraint functions. Trust regions control the size of the steps taken in the inner-point phase, ensuring that the iteration remains close to the feasible regions. The algorithm re-adjusts the limit parameter and confidence zone radius to a solution. d) Trust-region SQP (Sequential et al.) Trust-region SQP is an extension of the Trust-region method for solving nonlinear optimization problems with equality and inequality constraints. It incorporates a sequential quadratic software design framework, which uses a trust-region that formulates the difficulty as a series of quadratic sub-problems to be solved at each iteration to ensure that the steps remain within the possible section of the restraints explained in the constraints. e) Inexact Trust-region Methods: Inexact trust region approaches relax the requirement to explain the confidence domain subproblem efficiently and allow for solutions. This can be useful when the sub-problem is computationally expensive or does not require high accuracy. Incorrect methods lead to stagnation in solving sub- problems and allow the solver to interrupt them before achieving higher accuracy. This can result in significant computational savings. 4. The Applications, Numerical Simulations will be Preferred Generally, the Trust-region approach has been successfully applied in various fields and has proven effective in solving optimization problems. Here are a few examples of its use in mathematical simulations. 4.1. Machine Learning Trust-region methods are widely used in training machine learning models, especially in optimizing deep learning. It can be used to optimize neural network parameters such as weights and biases to reduce loss functions. The trust-region method helps to navigate the parameter space with higher dimensions efficiently and converge to the optimal solution. Numerical simulation: Consider a scenario where the Trust-region method trains deep neural networks for image classification. The algorithm fine-tunes the masses and biases of the network to lessen the loss of cross-entropy. Through iterations, the Trust-region method can obtain optimized parameters of accurate image classification. 4.2. Computer Vision: Trust-region methods find utility in computer vision tasks, such as image reconstruction and camera position estimation. These tasks typically require solving nonlinear optimization problems to reconstruct images, retrieve missing information, or reconstruct 3D models. Confidence-space methods can provide robust and accurate solutions in such cases. Numerical Simulation: Assume that the Trust-region 27 method estimates the camera's position from a coherent 2D image. By optimizing the pose parameters, the algorithm accurately matches the identified image points with the observations, resulting in an accurate camera position estimate (Conn et al., 2020). 4.3. Engineering Design Optimization Trust-region approaches are used in engineering design optimization problems, where the objective is to find optimal design parameters that satisfy a performance criterion or objective. These problems typically involve nonlinear objective functions, constraints, and designs. There are also variables. The trust-region approach helps identify optimal solutions by thoroughly analyzing the design space. Numerical simulation: Consider the example of the optimization of an airplane wing. The Trust-region method is used to find optimal size parameters, such as airfoil thickness distribution, that reduce drag while meeting structural constraints. The algorithm iteratively adjusts size parameters within the trust region, converging to an optimal design that improves aerodynamic performance. 4.4. Robust Control Trust-region methods are applied to complex control problems to develop a control system that can handle uncertainty and chaos. These problems involve optimizing control parameters to achieve the desired system performance and ensure robustness against changes in system dynamics or external disturbances. Numerical Simulation: Assume that a complex controller for an autonomous vehicle is designed using the Trust-region method. The algorithm optimizes the control parameters to minimize the cost work representing the trajectory tracking performance while considering vehicle dynamics and external disturbance uncertainty. Through iterations, the Trust-region method obtains rugged control parameters, ensuring stable and precise vehicle control. 4.5. Comparative Analysis Trust-region and line search approaches are popular techniques for resolving optimization complications. Although they aim to find an optimal solution, they have different ways of determining the step size and direction. Here is a crucial difference between trust region together with the line search methods. 5. Step Size Determination The trust-region approach determines the step size by solving a sub problem in the trust region. The confidence region limits the maximum distance of the procedure from the current iteration. The step size is obtained by reducing the local sample of the objective purpose in the trust region, subject to a constraint (Conn et al., 2020). The size of the trust region is accustomed dynamically based on the arrangement between the prototypical and the definite performance values. On the other hand, text search methods perform a one- dimensional search on a particular search path and specify the step size. The search direction is usually calculated using gradient information. The step size is chosen based on a line- finding procedure that optimizes the scalar function in the search path. Standard line-finding methods include Armijo's rule, Wolf's conditions, and Goldstein's conditions. 5.1. Global versus Local Information The trust-region approach uses global information about the objective function, such as gradient and possibly Hessian, to build local models and make informed decisions about step size and direction. Line recognition methods mainly rely on spatial information such as gradient to determine the step shape and direction. They focus on finding an appropriate step size in a search strategy that satisfies certain conditions, such as sufficient or extensive defects. Line search methods do not consider the larger context or include global information about the objective function. 5.2. Convergence Properties The Trust-Region method is known for its robustness and ability to solve nonconvex worst-case, and constrained optimization problems. The balance between exploration and implementation, compliance with step size, and optimization of the confidence zone leads to the optimization state. Depending on the problem, it can converge to local minima, fixed points, or global minima. Line search methods are effective for convex problems or problems where the aim function is even and consistent. Convergence to a certain point is usually guaranteed for convex functions. However, they may need help with nonconvex problems or poor-quality functions since they rely on local information and adhere to poor local minima. 5.3. Computational Cost The trust-region method typically requires solving a sub- problem in the confidence region at each iteration, which can be computationally expensive. The sub-problem involves solving a quadratic or nonlinear optimization problem. However, modern confidence-domain methods often use efficient methods, such as conjugate gradient methods or approximate Hessian statistics, to reduce computational cost Line search methods typically have lower computational costs because they require a one-dimensional search only in the search path. Line searches can be done efficiently with simple rules or conditions. However, more iterations may be required to achieve convergence when relying on local information and not considering the overall behavior of the objective function (Conn et al., 2020). 6. Conclusion In conclusion, when calculating the step size and direction, the Trust-Region together with the line search methods take two entirely different approaches. While line search methods concentrate on local information and optimize more significant steps in the search direction, trust-region approaches take a global perspective, develop local models, and dynamically adjust trust regions (Yuan, 2019). References [1] Conn, A. R., Gould, N. I., & Toint, P. L. (2020). Trust region methods—society for Industrial and Applied Mathematics. [2] Yuan, Y. X. (2020). A review of trust region algorithms for optimization. In Iciam (Vol. 99, No. 1, pp. 271-282). [3] Yuan, Y. X. (2019). Recent advances in trust region algorithms. Mathematical Programming, pp. 151, 249–281. [4] Moré, J. J. (2020). Recent developments in algorithms and software for trust region methods (pp. 258–287). Springer Berlin Heidelberg. [5] Nocedal, J., & Wright, S. J. (2002). Trust-region methods. Numerical Optimization, 66-100.