




































©2025 Ada Academica https://adac.eeEur. J. Math. Anal. 5 (2025) 9doi: 10.28924/ada/ma.5.9
Numerical Results for Gauss-Seidel Iterative Algorithm Based on Newton Methods for

Unconstrained Optimization Problems

Nguyen Dinh Dung
TNU-University of Information and Communication Technology, Thai Nguyen, Vietnam

nddung@ictu.edu.vn

Abstract. Optimization problems play a crucial role in various fields such as economics, engineering,and computer science. They involve finding the best value (maximum or minimum) of an objectivefunction. In unconstrained optimization problems, the goal is to find a point where the function’svalue reaches a maximum or minimum without being restricted by any conditions. Currently, thereare many different methods to solve unconstrained optimization problems, one of which is the Newtonmethod. This method is based on using a second-order Taylor series expansion to approximate theobjective function. By calculating the first derivative (gradient) and second derivative (Hessian matrix)of the function, the Newton method determines the direction and step size to find the extrema. Thismethod has a very fast convergence rate when near the solution and is particularly effective forproblems with complex mathematical structures. In this paper, we introduce a Gauss-Seidel-typealgorithm implemented for the Newton and Quasi-Newton methods, which is an efficient approachfor finding solutions to optimization problems when the objective function is a convex functional. Wealso present some computational results for the algorithm to illustrate the convergence of the method.

1. Introduction
In this paper, we focus on solving the unconstrained nonlinear optimization problem

min
x∈Rn

f (x) (1)
where f (x) is a convex functional with second derivative on Rn. Optimization problem (1) is alsoa problem derived from many problems in different fields in economics and engineering. Solvingproblem (1) can lead to solving a system of nonlinear equations and has many different appli-cations, for example in solving the `1-norm problem arising from compressing sensing [1]- [4], invariational inequalities problems [5]- [6], and optimal power flow equations [7] among others. Ina broader sense, optimization should be understood as the activities aimed at obtaining the bestresult under certain conditions (maximizing profit, minimizing costs). The theory of optimizationmethods is not new, there are a huge number of optimization methods: methods based on the use

Received: 27 Oct 2024.
Key words and phrases. Convex optimization; Newton; Quasi-Newton; Hessen matrix; Gauss–Seidel.1

https://adac.ee
https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 2of Lagrange multipliers, methods of dynamic programming, and methods of the calculus of vari-ations, linear and nonlinear programming methods, there are currently many different solutionsdepending on the objective function. One of the simplest methods is the steepest descent methodor also known as the Gradient descent method [2], [8], this method is simple and applicable to afairly wide class of objective functions, the content of the method is to give a sequence of iter-ations x (k+1) = x (k) − αk∇f (xk), αk > 0, where αk is the step length determined by Armijo’srule after the exact or inexact line search, this method has the disadvantage of linear convergencerate. We want to improve the efficiency of the algorithm’s convergence, the Newton method is agood choice [9]- [12], Newton’s method was first proposed by Isaac Newton in 1964 when findingsolutions to nonlinear equations. To date, Newton’s methods have widely been used for solvingthe unconstrained nonlinear optimization problem. As a result, studies of Newton’s method forman extremely active area of research, with new variants being constantly developed and tested.Basic results on Newton’s method and comprehensive lists of references can be found, e.g., in thebooks by Dennis and Schnabel [13], Ostrowski [14], Ortega and Rheinboldt [15], Deuflhard [16] andCorless and Fillion [17], survey of Newton’s method in [18]. The general iterative rule for solving (1)starts from an initial approximation and generates a sequence using the general iterative scheme
x(k+1) = x(k) + αkdk . (2)

Where dk is an appropriate search direction. General class of algorithms of the form (2) is knownas the line search algorithms. The method has a local quadratic convergence, thus convergingextremely fast in a neighbourhood of the solution [19]. Nowadays, this method is extended tofind optimal solutions for multivariable functions based on Taylor expansion, the solution of theoptimization problem is performed in an iterative sequence x(k+1) = x(k)−
[
∇2f (x(k))

]−1∇f (x(k)),if the objective function is not quadratic, the above iteration sequence may diverge or converge to alocal minimum or converge to a saddle point. A variant of Newton’s method introduced in [20] is thegeneralized Newton’s method, in which the solution to the optimization problem is computed by theiteration sequence x(k+1) = x(k) − αk
[
∇2f (x(k))

]−1∇f (x(k)), where αk is called the step lengthand is determined by one-dimensional search methods in the direction − [∇2f (x(k))]−1∇f (x(k)). However, in some practical problems when leading to optimization problems where the objectivefunction does not have a second derivative, applying Newton’s method is not feasible. To overcomethis limitation, recently some results in [21] and [22] have proposed a quasi Newton algorithm forfinding solutions to nonlinear optimization problems, showing the effectiveness of the method.The numerical results in the papers all use Jacobi iteration, so we hope to improve the conver-gence of the algorithm by applying the idea of Gauss-Seidel iteration to the implementation ofNewton and quansi-Newton methods. So, in this paper, we propose Gauss – Seidel algorithmsimplemented for the Newton and quasi-Newton method for finding solutions at each iterationstep, in which we inherit the information of the component solutions calculated in the current

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 3iteration instead of using the solutions calculated in the previous iteration. The computationalresults illustrating the algorithm are given to confirm the convergence of the algorithm. Thepaper is organized as follows. Section 2 presents the Newton’s method and implementation ofthe Gauss-Seidel iterative algorithm based on Newton and Quasi-Newton methods. Experimentalresults illustrating the convergence are presented in Section 3. Finally, there are conclusions andreferences.
2. Proposed method

2.1. Newton’s method.Let x∗ is the minimum point of the functional, then the necessary condition is ∂f
∂xi
(x∗) = 0. inorder to determine the iterative sequence, we use the Taylor expansion for f (x), we have

f (x) = f (x(k)) +∇fk(x− x(k)) + 1
2
(x− x(k))2Jk , (3)

where ∇f (x) = ( ∂f∂x1 , ∂f∂x2 , ..., ∂f∂xn), Jk is the Hessen matrix and is defined as

∇2f (x) =


∂2f
∂x21

∂2f
∂x1∂x2

... ∂2f
∂x1∂xn

∂2f
∂x2∂x1

∂2f
∂x22

... ∂2f
∂x2∂xn

..........................................

∂2f
∂xn∂x1

∂2f
∂xn∂x2

... ∂2f
∂x2n

 .
Since 2, we have

∇f = ∇fk + (x − x (k))Jk . (4)
So, we have the iterative process.

x(k+1) = x(k) − (Jk)−1∇fk , k = 1, 2, 3, ..., n. (5)
Formula (4) is called Newton’s iteration formula with second-order convergence rate. The algo-rithm is implemented as follows:
Algorithm 1:
function x=newton(x(1),ε);
k=1;
while(‖∇fk‖ > ε)

d(k) = −(Jk)−1∇fk ;
x(k+1) = x(k) + d(k);
k=k+1;

end;
x = x(k);

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 4

According to this algorithm, at the step k : d(k) = −(Jk)−1∇fk , x (k+1) = x (k)+ d (k), we implementcomponent inheritance x(k+1)i calculated to calculate x (k+1)j , j = i + 1, ..., n, So (5) is replaced by
x
(k+1)
i = x

(k)
i + d

(k)
i , (6)

where
d
(k)
1 = −

n∑
j=1

[
(Jk)

−1]∇f (x (k)j )
d
(k)
i = −

i−1∑
j=1

[
(Jk)

−1]∇f (x (k+1)j )−
n∑

j=i+1

[
(Jk)

−1]∇f (x (k)j ), i = 2, ..., n
So, Newton’s algorithm is updated as follows:
Algorithm 2:
function x=newton(x(1),ε);
k=1;
n=length(x(0));
while(‖∇fk‖ > ε)

d(k) = 0;
for j=1:n
d
(k)
1 = d

(k)
1 −

[
(Jk)−1

]
∇f (x (k)j )

end;
x
(k+1)
1 = x

(k)
1 + d

(k)
1

for i=2:n
for j=1:i-1
d
(k)
i = d

(k)
i −

[
(Jk)−1

]
∇f (x (k+1)j )

end;
for j=i:n
d
(k)
i = d

(k)
i −

[
(Jk)−1

]
∇f (x (k)j )

end;
x
(k+1)

i = x
(k)

i + d
(k)

i

end;
k=k+1;

end;
x = x(k+1)In case the objective function is not second-order differentiable, then instead of using Newton’smethod, we will use the quasi-Newtonian method.

2.2. Quasi-Newton method.The idea of the quasi-Newton method [22] is derived from formula (5), we approximate the Hessen

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 5matrix Jk by matrix Bk So, since (3), we have the following iteration process:
x(k+1) = x(k) − αk [Bk ]−1∇fk , k = 1..n, (7)

where, αk is the step length determined in the direction Sk = − [Bk ]−1∇fk , it can change at eachiteration and satisfy the Wolfe condition [22].
f (x(k) + αkSk) ≤ f (x(k)) + c1αkSTk ∇f (x(k))
−STk ∇f (x(k) + αkSk) ≤ −c2STk ∇f (x(k))
0 < c1 < c2 < 1

(8)
αk is determined by algorithm 3

Algorithm 3:
function αk=LineSearch(f,xk , Sk );Initialize constants c1, c2, β satisfy 0 < c1 < c2 < 1; 0 < β < 1

α = α0

while(f (x (k) + αSk) > f (x (k)) + c1αS
T
k ∇f (x (k)) or −STk ∇f (x (k) + αSk) > −c2STk ∇f (x (k)))

α = βα;
end;
αk = α;In case the objective function is a convex function, the obtained optimal point is the global optimalsolution of problem (1). It is easy to see in [13], if Bk = I then the iterative formula (7) is thesteepest descent method published in [23]. Bk is an approximation matrix for the Hessen matrixand satisfies the condition

∇f (x(k)) = ∇f (x(k−1))− Bk(x(k) − x(k−1)). (9)
At x (k+1), we have

∇f (x(k+1)) = ∇f (x(k))− Bk+1(x(k+1) − x(k)) (10)
Or can write

Bk+1dk = gk , (11)
where,

dk = x
(k+1) − x (k), gk = ∇fk+1 −∇fkFormula (11) can be rewritten as follows:

dk = [Bk+1]−1 gk , (12)
where, Bk+1 is a positive definite symmetric matrix and is updated according to the formula

Bk+1 = Bk + CZZT . (13)

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 6

Since (11), we have (Bk + CZZT
)

dk = gk . So, CZ = gk−[Bk ]dk
ZT dk

. Let Z = gk − [Bk ] dk , then
C = 1

ZT dk
and

Bk+1 = Bk +
(gk − Bkdk) (gk − Bkdk)T

(gk − Bkdk)T dk
. (14)

We can also use the following calculation: Bk+1 = Bk + C1Z1ZT1 + C2Z2ZT2 , dk = Bkgk +
C1Z1(ZT1 gk) + C2Z2(ZT2 gk). Let Z1 = dk and Z2 = Bkgk , similar to formula (15), we have

Bk+1 = Bk +
gkgTk
gTk dk

−
(Bkdk) (Bkdk)T

dTk Bkdk
(15)

Thus, the algorithm to find the solution of problem (1) is implemented as follows:
Algorithm 4:
function x=QNewton(x(1), ε);
k=1
Bk = I;
while(‖∇fk‖ > ε)

Sk = − [Bk ]−1∇fk ;
αk= LineSearch(f , x (k),Sk );
x(k+1) = x(k) + αkSk ;
dk = x(k+1) − x(k);
gk = ∇fk+1 −∇fk ;Update Bk+1 according to (14) or (15);
k=k+1;

end;
x = x(k+1);According to this algorithm, starting from point x(1), The iteration sequence (15) converges to thelocal optimum x∗ and satisfied.

f (x(1)) ≥ ... ≥ f (x(k)) ≥ ... ≥ f (x∗)

In case the objective function is a convex function, the obtained optimal point is the global optimalsolution of problem (1). To illustrate the theoretical results, here are some experimental calculationresults of the algorithm.
3. Experimental results and discussions

In this section, we perform experimental calculations to illustrate the convergence of the algorithmintroduced in the paper. The data is given:Objective function
f (x) =

10∑
i=1

(xi − i)4 (16)

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 7Initial approximation: x(1) = (0, 0, ..., 0). It is easy to see that the exact solution of problem (1)is x∗ = (1, 2, ..., 10), f (x∗) = 0. The objective function (16) is differentiable at all levels, so theHessen matrix exists, so we can completely apply Newton’s algorithm to solve problem (1). Let
er r =

∥∥x(k) − x∗
∥∥
2
, we have the computational results illustrating the convergence of Newton’salgorithm given in Table 1:

Table 1. Approximate solution of problem (1) obtained from Algorithm 2

x(k) k = 5 k = 10 k = 15 k = 20

x
(k)
1 0.8025 0.9740 0.9966 0.9995
x
(k)
2 1.6049 1.9480 1.9931 1.9991
x
(k)
3 2.4074 2.9220 2.9897 2.9986
x
(k)
4 3.2099 3.8960 3.9863 3.9982
x
(k)
5 4.0123 4.8699 4.9829 4.9977
x
(k)
6 4.8148 5.8439 5.9794 5.9973
x
(k)
7 5.6173 6.8179 6.9760 6.9968
x
(k)
8 6.4198 7.7919 7.9726 7.9964
x
(k)
9 7.2222 8.7659 8.9692 8.9959
x
(k)
10 8.0247 9.7399 9.9657 9.9955
err 3.8758 0.5104 0.0672 0.0089

The calculation results in Table 1 show that the approximate solution found converges to the exactsolution of the problem according to the number of iterations. The graphs in Figure 1 and Figure2 illustrate the convergence of the algorithm.

Figure 1. Error graph according to the number of iterations with the number of iterations
k=1,2,. . . ,20 obtained from Algorithm 2

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 8

Figure 2. Objective function graph according to the number of iterations (number of iterations
k=1,2,. . . ,20) obtained from Algorithm 2From Figure 1 and Figure 2, it can be seen that the error function and the objective function aremonotonically decreasing functions with the number of iterations, which shows that the approximatesolution converges to the exact solution of Problem (1). Now, let us consider the following objectivefunction:

f (x) =


10∑
i=1

(
xi − 1i

)4 ∃xi < 1
i

10∑
i=1

(
xi − 1i

)√
xi − 1i ∀xi ≥ 1

i

(17)
The objective function does not have a second derivative at x∗ = (1, 12 , ..., 110), So, in order to findthe solution for Problem (1) with objective function (16), we perform Algorithm 4 with the Hessenmatrix approximation. Matrix updated according to formula (15). The calculation results are givenin Table 2.

Table 2. Approximate solution of problem (1) obtained from Algorithm 4

x(k) k = 5 k = 10 k = 15 k = 20

x
(k)
1 0.4598 0.8654 0.9708 1.0128
x
(k)
2 0.5080 0.5079 0.5078 0.5078
x
(k)
3 0.2322 0.3231 0.3459 0.3548
x
(k)
4 0.1490 0.2424 0.2659 0.2748
x
(k)
5 0.0987 0.1791 0.1997 0.2078
x
(k)
6 0.0672 0.1341 0.1525 0.1600
x
(k)
7 0.0471 0.1025 0.1196 0.1271
x
(k)
8 0.0340 0.0798 0.0959 0.1039
x
(k)
9 0.0251 0.0632 0.0784 0.0866
x
(k)
10 0.0190 0.0508 0.0649 0.0733
err 0.6032 0.1680 0.0722 0.0584

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 9The calculation results in Table 2 show that the approximate solution found converges to the exactsolution of the problem depending on the number of iterations. The calculation results show thatthe quasi-Newton method has the advantage of not requiring a quadratic differentiable objectivefunction, but the convergence is quite slow compared to the Newton method. The error functionand the objective function are not monotonically decreasing functions with the number of iterations,but tend to decrease gradually, which also confirms that the approximate solution converges to theexact solution of the problem. The graphs in Figure 3 and Figure 4 illustrate the convergence ofthe algorithm.

Figure 3. Error graph depending on the number of iterations with the number of iterations
k=1,2,. . . ,20 obtained from Algorithm 4

Figure 4. Graph of the objective function depending on the number of iterations (number of
iterations k=1,2,. . . ,20) obtained from Algorithm 4

https://doi.org/10.28924/ada/ma.5.9


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 104. Conclusion
In this paper, we implement an iterative algorithm to solve the unconstrained convex optimizationproblem based on Newton and Quasi-Newton iteration methods, in which the information of thecomponent solutions calculated in the current iteration is inherited instead of using the solutionscalculated in the previous iteration. The computational results according to the algorithm areperformed on the Matlab 2014 environment, the numerical results have confirmed the convergenceof the method and are consistent with the theory presented in the paper.

References
[1] S. Aji, P. Kumam, A.M. Awwal, M.M. Yahaya, W. Kumam, Two Hybrid Spectral Methods With Inertial Effect for Solv-ing System of Nonlinear Monotone Equations With Application in Robotics, IEEE Access 9 (2021) 30918–30928.

https://doi.org/10.1109/ACCESS.2021.3056567.[2] Y. Zhou, Y. Wu, X. Li, A New Hybrid PRPFR Conjugate Gradient Method for Solving Nonlinear Monotone Equationsand Image Restoration Problems, Math. Probl. Eng. 2020 (2020) 6391321. https://doi.org/10.1155/2020/
6391321.[3] M. Eshaghnezhad, S. Effati, A. Mansoori, A Neurodynamic Model to Solve Nonlinear Pseudo-Monotone ProjectionEquation and Its Applications, IEEE Trans. Cybern. 47 (2017) 3050–3062. https://doi.org/10.1109/TCYB.
2016.2611529.[4] S. Crisci, M. Piana, V. Ruggiero, M. Scussolini, A Regularized Affine-Scaling Trust-Region Method for ParametricImaging of Dynamic PET Data, SIAM J. Imaging Sci. 14 (2021) 418–439. https://doi.org/10.1137/20M1336370.[5] J.K. Liu, X.L. Du, M. Scussolini, A Gradient Projection Method for the Sparse Signal Reconstruction in CompressiveSensing, Appl. Anal. 97 (2018) 2122–2131. https://doi.org/10.1080/00036811.2017.1400510.[6] A.M. Awwal, L. Wang, P. Kumam, H. Mohammad, W. Watthayu, A Projection Hestenes–stiefel Method With SpectralParameter for Nonlinear Monotone Equations and Signal Processing, Math. Comput. Appl. 25 (2020) 27. https:
//doi.org/10.3390/mca25020027.[7] B. Ghaddar, J. Marecek, M. Mevissen, Optimal Power Flow as a Polynomial Optimization Problem, IEEE Trans.Power Syst. 31 (2016) 539–546. https://doi.org/10.1109/TPWRS.2015.2390037.[8] H.B. Curry, The Method of Steepest Descent for Non-Linear Minimization Problems, Quart. Appl. Math. 2 (1944)258–261. https://doi.org/10.1090/qam/10667.[9] D. Kovalev, K. Mishchenko, P. Richtárik, Stochastic Newton and Cubic Newton Methods With Simple Local Linear-Quadratic Rates, arXiv preprint arXiv:1912.01597 (2019). https://arxiv.org/abs/1912.01597.[10] R. Weerakoon, T.G.I. Fernando, A New Variant of Newton’s Method Based on the Rectangular Rule for SolvingNonlinear Equations, Appl. Math. Lett. 13 (2000) 87–93. https://doi.org/10.1016/S0893-9659(00)00112-1.[11] A. Forsgren, P.E. Gill, M.H. Wright, Interior Methods for Nonlinear Optimization, SIAM Rev. 44 (2002) 525–597.
https://doi.org/10.1137/S0036144502414942.[12] R. Fletcher, Practical Methods of Optimization, Second Edition, John Wiley and Sons, Chichester, (2000), 40-46.[13] J.E. Dennis Jr., R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, SIAMClassics in Applied Mathematics, SIAM, Philadelphia (1996). https://doi.org/10.1137/1.9781611971200.[14] A.M. Ostrowski, Solution of Equations and Systems of Equations, Academic Press, Basel (1960).[15] J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, NewYork (1970).

https://doi.org/10.28924/ada/ma.5.9
https://doi.org/10.1109/ACCESS.2021.3056567
https://doi.org/10.1155/2020/6391321
https://doi.org/10.1155/2020/6391321
https://doi.org/10.1109/TCYB.2016.2611529
https://doi.org/10.1109/TCYB.2016.2611529
https://doi.org/10.1137/20M1336370
https://doi.org/10.1080/00036811.2017.1400510
https://doi.org/10.3390/mca25020027
https://doi.org/10.3390/mca25020027
https://doi.org/10.1109/TPWRS.2015.2390037
https://doi.org/10.1090/qam/10667
https://arxiv.org/abs/1912.01597
https://doi.org/10.1016/S0893-9659(00)00112-1
https://doi.org/10.1137/S0036144502414942
https://doi.org/10.1137/1.9781611971200


Eur. J. Math. Anal. 10.28924/ada/ma.5.9 11

[16] P. Deuflhard, Newton Methods for Nonlinear Problems: Affine Invariance and Adaptive Algorithms, 2nd edition,Springer, Berlin (2011). https://doi.org/10.1007/978-3-642-23899-4.[17] R.M. Corless, N. Fillion, A Graduate Introduction to Numerical Methods: From the Viewpoint of Backward ErrorAnalysis, Springer, New York (2013). https://doi.org/10.1007/978-1-4614-8453-0.[18] B.T. Polyak, Newton’s Method and Its Use in Optimization, Eur. J. Oper. Res. 181 (2007) 1086–1096. https:
//doi.org/10.1016/j.ejor.2005.06.076.[19] L.V. Kantorovich, Functional Analysis and Applied Mathematics, Uspekhi Mat. Nauk 3 (1948) 89–185. https:
//doi.org/10.1070/RM1948v003n06ABEH003995.[20] K. Geleta, R.K. Vasudeva, Quasi-Newton Line Search Algorithm for Solving Unconstrained Non-Linear Least SquareOptimization Problem, Int. J. Basic Appl. Sci. 10 (2021), 11-21.[21] I. Povalej, Quasi-Newton’s Method for Multiobjective Optimization, J. Comput. Appl. Math. 255 (2014) 765–777.
https://doi.org/10.1016/j.cam.2013.06.045.[22] K. Geleta, R.K. Vasudeva, Quasi-Newton Line Search Algorithm for Solving Unconstrained Non-Linear Least SquareOptimization Problem, Int. J. Basic Appl. Sci. 13 (2021) 9–18.[23] G. Yuan, S. Lu, Z. Wei, A Line Search Algorithm for Unconstrained Optimization, J. Softw. Eng. Appl. 3 (2010)503–509. https://doi.org/10.4236/jsea.2010.35057.

https://doi.org/10.28924/ada/ma.5.9
https://doi.org/10.1007/978-3-642-23899-4
https://doi.org/10.1007/978-1-4614-8453-0
https://doi.org/10.1016/j.ejor.2005.06.076
https://doi.org/10.1016/j.ejor.2005.06.076
https://doi.org/10.1070/RM1948v003n06ABEH003995
https://doi.org/10.1070/RM1948v003n06ABEH003995
https://doi.org/10.1016/j.cam.2013.06.045
https://doi.org/10.4236/jsea.2010.35057

	1. Introduction
	2. Proposed method
	2.1. Newton's method
	2.2. Quasi-Newton method

	3. Experimental results and discussions
	4. Conclusion
	References

