EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS 2025, Vol. 18, Issue 3, Article Number 6377 ISSN 1307-5543 – ejpam.com Published by New York Business Global An Enhanced Conjugate Gradient Method for Nonlinear Minimization Problems Ahmed Anwer Mustafa1,∗, Hussein Ageel Khatab2 1 Department of Mathematics, College of Education, University of Zakho, Kurdistan Region, Iraq 2 Department of Mathematics, College of Science, University of Zakho, Kurdistan Region, Iraq Abstract. Because of their computing efficiency and minimal memory requirements, conjugate gradient techniques are a fundamental family of algorithms for handling large-scale unconstrained nonlinear optimization problems. A new version of the Hestenes-Stiefel (HS) technique is presented in this study with the goal of improving convergence properties without compromising ease of use. We rigorously prove the global convergence qualities of the proposed approach under standard assumptions and show that it meets the conjugacy, descent, and adequate descent constraints. Numerous numerical tests, covering a wide range of benchmark issues, show that the suggested strategy routinely performs better than the traditional HS approach in terms of function evaluations and iteration count. 2020 Mathematics Subject Classifications: 65K05, 46N10, 90C26, 90C30 Key Words and Phrases: Nonlinear Optimization, Unconstrained Optimization, Conjugate Gradient Method, Descent Condition, Global Convergence 1. Introduction With applications in scientific computing, machine learning, and engineering design, nonlinear unconstrained optimization remains a fundamental topic of study in numerical optimization [1]. Iterative techniques like conjugate gradient (CG) algorithms work well because they can strike a compromise between computing economy and convergence guar- antees. Numerous alterations to traditional CG techniques have been put forth recently in an effort to boost efficiency and guarantee convergence under lax circumstances [2, 3]. Notably, three-term formulations have demonstrated potential for reducing storage needs and speeding up convergence [4]. Global convergence of these CG variations has been made easier by improvements in the line search algorithms, especially those that meet the strong Wolfe criteria [5]. Spectral ∗Corresponding author. DOI: https://doi.org/10.29020/nybg.ejpam.v18i3.6377 Email addresses: ahmed.mustafa@uoz.edu.krd (A. A. Mustafa), hussein.khatab@uoz.edu.krd (H. A. Khatab) https://www.ejpam.com 1 Copyright: © 2025 The Author(s). (CC BY-NC 4.0) A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 2 of 12 conjugate gradient techniques, which dynamically integrate curvature information into parameter updates, have also received fresh attention [6, 7]. Large-scale optimization situations, such as deep learning training procedures and sparse regression, have benefited from such advancements [8]. Furthermore, by carefully examining the adequate descent and conjugacy aspects of CG approaches, current research has shown their theoretical soundness [9, 10]. These advancements highlight the necessity of more research into hybrid or parameter-adaptive CG techniques that maintain desired convergence behavior in a variety of issue scenarios. This paper contributes to this ongoing research by introducing a modified CG method with demonstrable performance gains on classical benchmark problems. Consider the unconstrained optimization problem: min f(x), x ∈ Rn (1.1) where f : Rn → R is a real-valued, continuously differentiable function. A nonlinear conjugate gradient method generates a sequence {xk}, k ≥ 0, starting from an initial guess x0 ∈ Rn, using the recurrence xk+1 = xk + αkdk (1.2) where vk = xk+1 − xk, αk is positive step size and is obtained by some line searches, and dk is a search direction. The conjugate gradient method’s simplicity and extremely low memory requirements make it a potent line search technique for resolving large-scale optimization problems. The search direction dk of the conjugate gradient methods is given by: dk+1 = −gk+1 + βkdk, k ≥ 0, d0 = −g0 (1.3) where gk+1 = ∇f(xk+1) and βk is a scalar. Many formulas for βk are suggested, like Fletcher-Reeves (FR) [11], Dai and Yuan (DY) [12], Hestenes-Stiefel (HS) [13], Polak- Ribiere-Polyak (PRP) [14], conjugate descent (CD) [15], and Liu-Storey (LS) method [16], these formulas are as follows: βFR k = ∥gk+1∥2 ∥gk∥2 (1.4) βDY k = ∥gk+1∥2 dTk (gk+1 − gk) (1.5) βHS k = gTk+1(gk+1 − gk) dTk (gk+1 − gk) (1.6) βPRP k = gTk+1(gk+1 − gk) ∥gk∥2 (1.7) βCD k = ∥gk+1∥2 −dTk gk (1.8) A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 3 of 12 βLS k = gTk+1(gk+1 − gk) −dTk gk (1.9) where yk = gk+1 − gk, and the symbol ∥ · ∥ denotes the Euclidean norm of vectors. Also, many parameters are proposed: Hussein Ageel Khatab and Salah Gazi Sha- reef suggested a new conjugate gradient method for solving nonlinear unconstrained op- timization problems by using three terms conjugate gradient method [4]. Ahmed Anwer Mustafa [6] proposed a novel algorithm to perform spectral conjugate gradient descent for an unconstrained, nonlinear optimization problem. The same researcher suggested another algorithm, see [7]. The properties of conjugate gradient methods have been explored greatly through their global convergence properties. Global convergence results for the FR method without regular restarts and in exact line search have been obtained by Zoutendijk [17], and by Al-Baali [18] in connection with inexact line searches. Dai and Yuan [12] shown that the DY method is descent and globally convergent if the Wolfe line search f(xk + αkdk)− f(xk) ≤ c1αkg T k dk, gTk+1dk ≥ c2g T k dk is used. The structure of this paper is as follows: In Section 2, a new conjugate gradient method will be proposed. Its conjugacy condition, descent condition, and sufficient descent condition will be proved in Section 3. In Section 4, its global convergence will be examined. Some numerical experiments about this new conjugate gradient method will be presented in Section 5, and the conclusion will be provided in Section 6. 2. New Conjugate Gradient Method and Its Algorithm In this section, we develop a modified nonlinear HS method for solving nonlinear unconstrained optimization problems. To obtain the generated sufficient descent direction, Hager and Zhang [19] showed a new conjugate gradient method (CG-C) obtained by modifying the HS method, which generates sufficient descent directions at each step, without relying on any line search. The scalar βk in CG-C method is determined by βNHS k = max { βN k , µk } (2.1) where βN k = gTk+1yk dTk yk − 2∥yk∥2gTk+1dk (dTk yk) 2 , µk = 1 ∥dk∥min{∥gk∥, µ} , and µ > 0 is a constant. With the weak Wolfe-Powell line search, Hager and Zhang [19] established a global convergence result for (2.1) when the objective function f(x) is a general nonlinear function. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 4 of 12 Also, Hussein Ageel Khatab and Salah Gazi Shareef [20] suggested the following con- jugate gradient method βNEW k = gTk+1yk dTk yk − gTk+1vk dTk yk − µ gTk+1dk dTk yk ∥gk∥2 gTk yk (2.2) where µ ∈ (0, 1). We suggest a new parameter as follows: βNew k = gTk+1yk dTk yk − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dk (2.3) where µ > 0. Algorithm of The New Conjugate Gradient Method (βNEW k ) Step 1: Select x0 and ε = 10−5. Step 2: Set d0 = −g0, gk = ∇f(xk), Set k = 0. Step 3: Compute the step length αk > 0 satisfying the Wolfe line search conditions f(xk + αkdk)− f(xk) ≤ c1αkg T k dk, |gTk+1dk| ≤ c2|gTk dk|, where 0 < c1 < c2 < 1. Step 4: Compute xk+1 = xk + αkdk, gk+1 = ∇f(xk+1), If ∥gk+1∥ ≤ ε, then stop. Step 5: Compute βNew k by equation (2.3). Step 6: Compute dk+1 = −gk+1 + βNew k dk. Step 7: If |gTk+1gk| > 0.2∥gk+1∥2, then go to Step 2. Otherwise, k = k + 1, and go to Step 3. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 5 of 12 3. Conjugacy Condition, Descent Condition, and Sufficient Condition In this section, we will prove the Conjugacy condition, Descent Condition, and Suffi- cient of the new method. Theorem 1: - Assume that the sequence {xk} is generated by (1.2), then the new method (2.3) satisfies the conjugacy condition, i.e. dTk+1yk = t gTk+1vk, where t > 0. Proof: - By putting (2.3) in equation (1.3) to get dk+1 = −gk+1 + βnew k dk or, dk+1 = −gk+1 + ( gTk+1yk dTk yk − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dk ) dk (2.4) Multiply both sides of equation (2.4) by yk from right-hand side, we get dTk+1yk = −gTk+1yk + gTk+1yk dTk yk dTk yk − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dkd T k yk or, dTk+1yk = −µ ∥gk+1∥2∥vk∥2∥xk+1∥ dTk yk gTk+1dk (2.5) implies that dTk+1yk = − ( µ 1 αk ∥gk+1∥2∥vk∥2∥xk+1∥ dTk yk ) gTk+1vk (2.6) Suppose that t = ( µ 1 αk ∥gk+1∥2∥vk∥2∥xk+1∥ dTk yk ) then, dTk+1yk = −t gTk+1vk Theorem 2: - Assume that the sequence {xk} is generated by (1.2), then the search direction (1.3) of the new method (2.3) satisfies the descent condition, i.e. dTk+1gk+1 ≤ 0 with exact and inexact line search. Proof: - We will prove by mathematical induction. If k = 0, then, d0 = −g0, dT0 g0 = −gT0 g0 = −∥g0∥2 ≤ 0, We suppose that dTk gk ≤ 0. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 6 of 12 Now, we prove the case k+ 1. From (1.3) and (2.3), we have (2.4), multiply both sides of equation (2.4) by gk+1 from the right-hand side, we get dTk+1gk+1 = −gTk+1gk+1 + gTk+1yk dTk yk dTk gk+1 − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dkd T k gk+1 (2.7) If the step length αk is chosen by an exact line search which requires dTk gk+1 = 0, then the proof is complete. If the step length αk is chosen by inexact line search which requires dTk gk+1 ̸= 0, then the first two terms on the right-hand side of equation (2.7) are less than or equal to zero since HS satisfies the descent condition. The third term is less than or equal to zero, so we get dTk+1gk+1 ≤ −µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 (gTk+1dk) 2 (2.8) Then, dTk+1gk+1 ≤ 0. Theorem 3: - Assume that the sequence {xk} is generated by (1.2), then the search direction (1.3) with the new method (2.3) satisfies the sufficient descent condition, i.e. dTk+1gk+1 ≤ −C∥gk+1∥2. Proof: - We multiply the right-hand side of inequality (2.8) by ∥gk+1∥2 ∥gk+1∥2 , we get dTk+1gk+1 ≤ −µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 (gTk+1dk) 2 ( ∥gk+1∥2 ∥gk+1∥2 ) (2.9) Suppose that C = µ ∥gk+1∥2∥vk∥2∥xk+1∥ ∥gk+1∥2(dTk yk)2 (gTk+1dk) 2 Then, dTk+1gk+1 ≤ −C∥gk+1∥2 4. Convergence Analysis Assume that: • The level set S = {x ∈ Rn : f(x) ≤ f(x0)} is bounded. • In a neighborhood N of S, the function f is continuously differentiable and its gradient is Lipschitz continuous, i.e., there exists a constant L > 0 such that ∥g(x)− g(y)∥ ≤ L∥x− y∥ for all x, y ∈ N. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 7 of 12 Under these assumptions on f , there exists a constant ρ ≥ 0 such that ∥g(x)∥ ≤ ρ, for all x ∈ S. In [5], it is proved that for any conjugate gradient method with a strong Wolfe line search the following general result holds: Lemma 1: Let assumption (i) and (ii) hold and consider any conjugate gradient method (1.3) and (2.3), where dk is a descent direction and αk is obtained by the strong Wolfe line search. If ∑ k≥1 1 ∥dk+1∥2 = ∞ (4.1) then, lim k→∞ ∥gk+1∥ = 0. (4.2) For a uniformly convex function which satisfies the above assumptions, we can prove that the norm of dk+1 given by (1.3) and (2.3) is bounded above. Assume that the function f is uniformly convex, i.e., there exists a constant δ ≥ 0, such that for all x, xk ∈ S (g(x)− g(xk)) T (x− xk) ≥ δ∥x− xk∥2. (4.3) and the step length αk is given by a strong Wolfe line search: f(xk + αkdk) ≤ f(xk) + c1αkg T k dk (4.4)∣∣∇f(xk + αkdk) Tdk ∣∣ ≤ c2|gTk dk| (4.5) Using Lemma 1 the following result can be proved. Theorem 4: Suppose that the assumptions (i) and (ii) hold. Consider the algorithm (2.3) and where µ > 0 and αk is obtained by a strong Wolfe line search. If dk tends to zero and there exist nonnegative constants η1 and η2 such that ∥gk∥2 ≥ η1∥vk∥2 and ∥gk+1∥2 ≤ η2∥vk∥ (4.6) and f is a uniformly convex function, then lim k→∞ gk+1 = 0 (4.7) Proof: Consider the new method βNew k = gTk+1yk dTk yk − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dk or, |βNew k | = | gTk+1yk dTk yk − µ ∥gk+1∥2∥vk∥2∥xk+1∥ (dTk yk) 2 gTk+1dk| A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 8 of 12 This implies that |βnew k | ≤ ∥gk+1∥∥yk∥ dTk yk + µ ∥gk+1∥2∥vk∥2∥xk+1∥ αk(d T k yk) 2 ∥gk+1∥∥vk∥ Here, |βnew k | ≤ ρL∥vk∥∥vk∥ δ∥vk∥2 + µ η2∥vk∥∥vk∥2∥xk+1∥ αk(δ∥vk∥2)2 ρ∥vk∥ or, |βnew k | ≤ ρL δ + µη2 ρ∥xk+1∥ αkδ Now, ∥dk+1∥ ≤ ∥gk+1∥+ 1 αk ( ρL δ + µη2 ρ∥xk+1∥ αkδ ) ∥vk∥ Implies that ∥dk+1∥ ≤ ρ+ 1 αk ( ρL δ + µη2 ρ∥xk+1∥ αkδ ) D1 Where D1 = {y−z}, y, z ∈ S is the diameter of the level set S. There exists a constant D2 such that ∥xk+1∥ ≤ D2, then ∥dk+1∥ ≤ ρ+ 1 αk ( ρL δ + µη2 ρD2 αkδ ) D1 Showing that (4.1) is true. By Lemma 1, it follows that (4.2) is true, which for uniformly convex functions is equivalent to (4.7). 5. Numerical Results We compared the new method with the conjugate gradient (HS) method in this section, the comparative tests involve Well-known nonlinear problems (standard test function) with different dimensions n = 10, 300, 1000, and 4000, all programs are written in FORTRAN90 language and for all cases the stopping condition is ∥gk+1∥ ≤ 10−5. The results given in Table 1 specifically quote the number of function NOF and the number of iteration NOI. More experimental results in Table 1 and the figures confirm that the new conjugate gradient method is superior to the standard conjugate gradient (HS) method concerning the NOI and NOF. The performance results are shown in Figures 1 and 2, using the performance profile introduced by Dolan and More (2002)[21]. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 9 of 12 Table 1: Comparative Performance of the Two Algorithms (HS and New Method) No Test Function N NOI (HS) NOF (HS) NOI (New) NOF (New) 1 Wood 10 30 68 30 68 300 30 68 26 60 1000 30 68 31 71 4000 30 68 24 57 2 OSP 10 13 58 13 58 300 88 290 79 251 1000 156 473 140 419 4000 230 700 228 678 3 Powell 10 38 108 39 119 300 40 122 34 109 1000 41 124 30 92 4000 41 124 44 115 4 Miele 10 31 102 31 102 300 40 146 40 147 1000 46 176 39 139 4000 54 211 35 122 5 G-Central 10 22 159 22 159 300 23 171 23 170 1000 23 171 22 164 4000 28 248 31 269 6 Shallow 10 8 21 8 21 300 8 21 8 21 1000 9 24 8 21 4000 9 24 8 21 7 Beal 10 11 28 11 28 300 12 30 11 28 1000 12 30 9 25 4000 12 30 13 33 8 Sum 10 6 34 6 34 300 19 105 19 105 1000 23 128 23 128 4000 32 142 31 128 9 Wolfe 10 32 65 32 65 300 48 97 44 89 1000 70 141 45 93 4000 169 355 55 115 10 Cubic 10 13 37 13 37 300 13 37 13 37 1000 13 37 13 37 4000 13 37 13 37 11 Non-Diagonal 10 26 72 26 72 300 29 79 30 81 1000 29 79 27 77 4000 F F F F 12 Rosen 10 30 83 30 83 300 30 83 30 83 1000 30 83 30 83 4000 30 83 30 83 13 TRI 10 F F 9 19 300 148 297 147 295 1000 288 577 288 577 4000 600 1201 600 1201 14 G-Dixon 10 21 45 21 45 300 508 1136 536 1206 1000 4005 8015 534 1163 4000 560 1217 505 1107 15 Fred 10 8 23 8 23 300 8 23 8 23 1000 8 23 8 23 4000 8 23 8 23 A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 10 of 12 Figure 1: Performance profile on the number of iterations Figure 2: Performance profile on the number of functions 6. Conclusion In this paper, we suggested a new conjugate gradient method for unconstrained opti- mization problems. We proved the conjugacy condition, descent condition, sufficient de- scent condition, and the global convergence of the new method. Implemented and tested to A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 11 of 12 some extent. The new method was compared with the standard conjugate gradient (HS) method. The numerical tests were carried out on low- and high-dimensionality problems and comparisons were made amongst different test functions with inexact line search. We used fifteen function with different dimensions (10, 300, 1000, and 4000). Two important measures are used to assess the approaches’ performance: the number of iterations (NOI) and the number of function evaluations (NOF). The numerical results in Table 1 and the figures 1 and 2 demonstrate that the proposed method performs effectively. References [1] I. Ahmed and B. Mohanty. Recent advances in nonlinear optimization methods for machine learning models. Applied Intelligence, 53:1587–1603, 2023. [2] R. Bakar and Z. J. Weng. An adaptive conjugate gradient method with descent property. Numerical Algorithms, 94:107–129, 2023. [3] H. Chen and J. Zhang. Three-term conjugate gradient algorithms with inexact line search for large-scale optimization. Optimization Letters, 16(8):2315–2333, 2022. [4] A. K. Hussein and S. G. Shareef. A new parameter conjugate gradient method based on three terms unconstrained optimization. General Letters in Mathematics, 7(1):39– 44, 2019. [5] Y. H. Dai, J. Y. Han, G. H. Liu, D. F. Sun, H. X. Yin, and Y. X. Yuan. Convergence properties of nonlinear conjugate gradient methods. SIAM Journal on Optimization, 10:348–358, 1999. [6] A. A. Mustafa. New spectral ls conjugate gradient method for nonlinear unconstrained optimization. International Journal of Computer Mathematics, 100(4):838–846, 2023. [7] A. A. Mustafa. A new algorithm for spectral conjugate gradient in nonlinear opti- mization. Mathematics and Statistics, 10(2):293–300, 2022. [8] Y. Liu and H. Wang. Spectral-type cg methods for deep learning optimization. Jour- nal of Computational Mathematics, 43(1):45–67, 2024. [9] A. K. Mohammed and S. R. Al-Taie. Global convergence of parameterized conjugate gradient algorithms. Mathematics, 12(3):389–404, 2024. [10] F. Zhang and C. Li. Sufficient descent and global convergence in nonlinear cg methods with modified β-updates. Journal of Optimization Theory and Applications, 195:623– 647, 2025. [11] R. Fletcher and C. M. Reeves. Function minimization by conjugate gradients. The Computer Journal, 7(2):149–154, 1964. [12] Y. H. Dai and Y. Yuan. A nonlinear conjugate gradient method with a strong global convergence property. SIAM Journal on Optimization, 10:177–182, 1999. [13] M. R. Hestenes and E. Stiefel. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 49(6):409–436, 1952. [14] E. Polak and G. Ribière. Note sur la convergence de méthodes de directions con- juguées. Revue Française d’Informatique et de Recherche Opérationnelle, 16:35–43, 1969. A. A. Mustafa, H. A. Khatab / Eur. J. Pure Appl. Math, 18 (3) (2025), 6377 12 of 12 [15] R. Fletcher. Practical Methods of Optimization. Wiley, New York, 2 edition, 1987. [16] Y. Liu and C. Storey. Efficient generalized conjugate gradient algorithms, part 1: Theory. Journal of Optimization Theory and Applications, 69:129–137, 1991. [17] G. Zoutendijk. Nonlinear programming, computational methods. In J. Abadie, editor, Integer and Nonlinear Programming, pages 37–86. North-Holland, 1970. [18] A. Al-Baali. Descent property and global convergence of the fletcher-reeves method with inexact line search. IMA Journal of Numerical Analysis, 5:121–124, 1985. [19] W. W. Hager and H. Zhang. A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM Journal on Optimization, 16(1):170–192, 2005. [20] S. G. Hussein and A. K. Hussein. General Letters in Mathematics, 9(2), 2020. [21] E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 91(2):201–213, 2002.