EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS
Vol. 15, No. 4, 2022, 1683-1693
ISSN 1307-5543 – ejpam.com
Published by New York Business Global

The Convergent Properties of a New Parameter for
Unconstrained Optimization

Ghada M. Al-Naemi

Department of Mathematics, Faculty of Computer Science and Mathematics,
University of Mosul, Mosul, Nineveh, Iraq

Abstract. Because of its simplicity, low memory requirement, low computational cost, and global
convergence properties, the Conjugate Gradient (CG) method is the most popular iterative math-
ematical technique for optimizing both linear and nonlinear systems. Some classical CG methods,
however, have drawbacks such as poor global convergence and numerical performance in terms of
iterations and function evaluations. To address these shortcomings, researchers proposed new CG
parameter variants with efficient numerical results and good convergence properties. We present a
new conjugate gradient formula βGh

k based on the memoryless self-scale DFP quasi-Newton (QN)
method in this paper. The proposed new formula fulfills the sufficient descent property and the
global convergent condition with any proposed line research. When the exact line search is used,
the proposed formula is reduced to the classical HS formula. Finally, we conclude that our proposed
method is effective.

2020 Mathematics Subject Classifications: 90C30, 65K05, 90C25, 90C26, 90C27

Key Words and Phrases: Conjugate gradient, self-scale DFP, strong Wolfe-Powell line search,
sufficient descent property

1. Introduction

For solving the unconstrained minimization problem, the quasi-Newton methods are
extremely useful and efficient to solve.

Min.z(x), x ∈ Rn (1)

where z : Rn → R is twice continuously differentiable [5]. Broyden [2] introduced the QN
family of variable metric formulas in 1970, which is the most efficient technique for mini-
mizing a non-linear function z(x). The following quasi-Newton equation has traditionally
been used to update the iterate matrix:

βk+1vk = yk (2)

DOI: https://doi.org/10.29020/nybg.ejpam.v15i4.4538

Email address: drghadaalnaemi@uomosul.edu.iq (G. M. Al-Naemi)

https://www.ejpam.com 1683 © 2022 EJPAM All rights reserved.



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1684

We have
vk = γkdk = xk+1 − xk, and yk = gk+1 − gk (3)

where γk > 0 is determined by a suitable line search. Iterative methods are commonly
used to solve (1), and the iterative formula is as follows:

xk+1 = xk + γkdk, k = 0, 1, 2, 3, . . . , (4)

If Hk is to be regarded as a close approximation to B−1
k , it follows that:

Hk+1yk = vk (5)

The direction dk is obtained by solving the equation the updating matrix Bk is required
to satisfy the equation (2) and the usual quasi-Newton equation (5).

dk = −Hkgk (6)

The nonlinear conjugate gradient (CG) method is one of the most well-known methods
for solving the unconstrained optimization problem (1), which is especially useful when
the dimension n of z(x) is large [7]. This is because the iteration is simple and requires
little memory. The search direction is typically defined as:

dk =

{
−gk, k = 0

−gk + βkdk−1, k ≥ 1
(7)

βk ∈ R, characterized the CG-method. If f(x) is a strictly convex quadratic function with
exact line search, the parameter βk is typically chosen to reduce the linear CG-method in
(4) and (7). [12][11][18][19][10][15][9] define the six pioneering forms of βk.

βHS
k =

gTk yk−1

yTk−1dk−1
; βFR

k =
gTk gk

gTk−1gk−1
; βPRP

k =
gTk yk−1

gTk−1gk−1
;

βCD
k =

gTk gk

yTk−1dk−1
; βLSk =

gTk yk−1

−gTk−1dk−1
; βDY

k =
gTk gk

yTk−1dk−1
;

Many of the classic parameters βk mentioned above have been modified by a group of
researchers, and another group has derived or imposed new parameters βk; however, not
all of them can be included in a research, for example, see [13][21][16][1][22].

To determine the convergence conditions of above methods, it is usually necessary that
the step size γk verify some properties, one of which is the strong Wolfe-Powel line search
(sWP):

f(xk + γkdk) ≤ f(xk) + ργkdk (8)

|g(xk + γkdk)
Tdk| ≤ σ|gTk dk| (9)

where 0 < σ < 0.5 < ρ < 1 are some fixed parameters. The step-size γk plays an essential
when investigating the sufficient descent condition



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1685

gTk dk < −ω||gk||2, ω > 0 (10)

and global convergence properties

lim
k→∞

||gk||2 = 0 (11)

2. New Formulas for β’s and its Algorithm

The DFP update was first proposed by Davidon, and popularized by Fletcher and
Powell. The DFP formula can be expressed as follows:

Hk =

[
Hk−1 +

vk−1v
T
k−1

yTk−1vk−1
−
Hk−1yk−1y

T
k−1Hk−1

yTk−1Hk−1yk−1

]
(12)

To scale the Hessian matrix Hk, we will use the self-scaling quasi-Newton method. Oren
[17] introduced self-scaling variable metric algorithms, which are defined as

ηk−1 =
vTk−1yk−1

||vk−1||2
(13)

which is a well-known and effective adaptive formula. Our proposed method’s general
strategy is to scale all DFP terms, i.e. update the matrix by self-scaling DFP of the form

Hk = ηk−1

[
Hk−1 +

vk−1v
T
k−1

yTk−1vk−1
−
Hk−1yk−1y

T
k−1Hk−1

yTk−1Hk−1yk−1

]
(14)

The preceding self-scaling DFP method is transformed into the memoryless self-scaling
DFP method when Hk is substituted for I (i.e. Hk ≡ I, where I is the identity matrix).
As a result, the memoryless DFP formed by:

Hk =
vTk−1yk−1

||vk−1||2

[
I +

vk−1v
T
k−1

yTk−1vk−1
−
yk−1y

T
k−1

yTk−1yk−1

]
(15)

To derive the new formula multiply both sides of equation (15) by (−gk), we have (6) and
in the CG (7), hence

−gk + βkdk−1 =
vTk−1yk−1

||vk−1||2

[
−gk +

vTk−1gk

yTk−1vk−1
vk−1 −

yTk−1gk

yTk−1yk−1
yk−1

]
(16)

When we multiply both sides of (16) by yk−1, we get

−gTk yk−1+βkd
T
k−1yk−1 =

vTk−1yk−1

||vk−1||2

[
−gTk yk−1 +

vTk−1gk

yTk−1vk−1
vTk−1yk−1 −

yTk−1gk

yTk−1yk−1
yTk−1yk−1

]



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1686

βkd
T
k−1yk−1 = gTk yk−1 −

vTk−1yk−1

||vk−1||2
vTk−1gk (17)

Use vk−1 = γk−1dk−1 in (17), we have

βk =
gTk yk−1

dTk−1yk−1
−

(
γk−1d

T
k−1yk−1

γ2k−1||dk−1||2

)(
γk−1d

T
k−1gk

dTk−1yk−1

)

βGh
k =

gTk yk−1

dTk−1yk−1
−

gTk dk−1

||vk−1||2
(18)

It should be noted that if we used exact line search βGh
k = βHS

k .
Now, we will go over the main steps of the algorithm that was used to create the new

formula of βGh
k

2.1. Algorithm A

Given x0 ∈ Rn, and ε > 0, set k = 0.

S1: Put dk = −gk, if ||gk|| < ε, stop, otherwise continue.

S2: Calculate γk by using (8) and (9).

S3: Calculate xk+1 by (4), and gk+1, if ||gk+1|| < ε, then stop; Otherwise continue.

S4: Calculate βGh
k by (19) and dk+1 by (7).

S5: If the restarting criteria |gTk gk−1| ≥ 0.2||gk||2 is satisfy, proceed to S1, else put
k = k + 1 go to S2.

2.2. Convergence Property

The following basic assumptions on the objective function are required to determine
the global convergence property for algorithm (A).

2.3. Assumption B

i. f(x) is constrained by the level set from below ψ = {x ∈ Rn, f(x) ≤ f(x0)}, where
x0 represents the starting point. Namely, there exists τ > 0 which implies ||xk|| ≤
τ ∀x ∈ ψ [3].

ii. f(x) it is a smooth in a specific neighborhood N of ψ, and its gradient is Lipschitz
continuous, i.e, there is a constant L greater than zero, such that

||∇f (x)−∇f (y) || ≤ L||x− y||, ∀x, y ∈ N (19)



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1687

Using algorithm (A), there is now a positive constant (ω), resulting in 0 < ||gk|| ≤ ω, ∀x ∈
ψ [14] To attain global convergence, algorithms (A) must be globally converged. To begin,
we will look into the new proposed method’s descent property.

Theorem 1. Impose that Assumptions (B) hold. Suppose the method of the form (4) and
(7) with βGh

k satisfy (18), and the step size γk satisfy sWP line search (8) and (9), then
there exists a constant ϑ > 0, s.t.

gTk dk ≤ −ϑ||gk||2, ϑ > 0, ∀k ≥ 0 (20)

Proof. To begin, the proof is trivial for k = 0, i.e.

d0 = g0 ⇒ gT0 d0 = −||g0||2

Multiplying both sides of (7) by gTk , we get

gTk dk = −gTk gk +

[
gTk yk−1

dTk−1yk−1
−

gTk dk−1

||dk−1||2

]
gTk dk−1

= −gTk gk +
gTk yk−1

dTk−1yk−1
gTk dk−1 −

(
gTk dk−1

)2
||dk−1||2

We know that gTk dk−1 ≤ dTk−1yk−1, and g
T
k dk−1 ≤ ||gk||.||dk−1||

gTk dk ≤ −||gk||2 +
gTk yk−1

dTk−1yk−1
dTk−1yk−1 −

(||gk||.||dk−1||)2

||dk−1||2

= −||gk||2 + gTk yk−1 − ||gk||2

gTk yk−1 = ||gk||2 − gTk gk−1

Using the restarting criteria, i.e. gTk gk−1 ≤ −0.2||gk||2, yield

gTk dk ≤ −2||gk||2 + ||gk||+ 0.2||gk||2

gTk dk ≤ −0.8||gk||2

As a result, (20) holds true for all k. After demonstrating that Algorithm (A) satisfies
the descent property, we must demonstrate Algorithm (A) global convergence under as-
sumption (B). We require the following lemmas, which are frequently used to demonstrate
global convergence and Zoutendijk [23] provides them.

Lemma 1. Let the Assumption (B) be correct. Assume any iteration method (4) and (7),
and γk obtained by the sWP (8) and (9). If∑

k≥1

1

||dk||2
= ∞ (21)

Then
lim inf
k→∞

||gk|| = 0 (22)



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1688

Theorem 2. Consider that Assumption (B) established. Suppose that the algorithm (A),
and γk is obtained through the sWP and dk is the descent direction. Then lim inf

k→∞
||gk|| = 0.

Proof. Because the descending property holds, we now have dk ̸= 0. Therefore, lemma
(2) is sufficient to show that ||dk|| is constrained above. From (2) and (8),

||dk|| =

∣∣∣∣∣
∣∣∣∣∣−gk +

[
gTk yk−1

dTk−1yk−1
−

dTk−1gk

||dk−1||2

]
dk−1

∣∣∣∣∣
∣∣∣∣∣

Since
∣∣dTk−1yk−1

∣∣ ≥ m||dk−1||.||yk−1||, where m > 0 [20], so

||dk|| ≤ ||gk||+
||gk||.||yk−1||

m||dk−1||.||yk−1||
− ||gk||

≤
(

1

m||dk−1||

)
||gk|| ≤

(
1

m.v

)
ω = µ

⇒
∑
k≥1

1

||dk||2
≥ 1

µ2

∑
k≥1

1 = ∞

As a result, (24) applies to all k.

3. Numerical Results

The main task in this section is to report the Algorithm (A performance)’s on a set of
test functions. All codes are written with double precision in FORTRAN. We chose twenty
unconstrained large-scale optimization test problems. We considered two experiments with
different numbers of variables (n=100 and 1000) for each test function. The test problems
are from the CUTE [6] library, as well as other large-scale optimization test problems from
[4]. To assess the reliability of our algorithms, we used the same test functions to compare
them to the well-known routines HS, DY, PRP and LS. All of these algorithms use sWP
(8) and (9) line searches with δ = 0.001 and σ = 0.5, respectively. When the following
stopping criterion is met ||gk+1|| ≤ 10−5, all of these methods terminate. Dolan and Mor’e
created performance profiling software. [8] was also used to analyze the execution Figures
1 and 2.



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1689

Table 1: Comparison between βHS , βPRP , βLS , and βGh with n = 100

Test functions βGh
k βHS

k βPRP
k βLSk

NOI NOF NOI NOF NOI NOF NOI NOF

Wood 27 64 33 73 29 67 30 69
Wolfe 54 101 55 103 57 114 55 115
Rosen 38 101 34 94 35 96 35 96
NON 28 67 30 78 32 81 31 80
Shallow 10 25 10 25 10 25 10 25
ENGVAL 1 21 44 22 46 24 46 22 46
Diagonal 2 54 207 60 213 58 214 60 221
Ex. BD1 20 40 22 46 19 39 23 48
Ex. Wood 27 61 29 65 32 67 29 65
Powell 31 92 37 104 39 112 41 118
Dixnaanc 5 15 5 15 5 15 5 15
DENSCHNF 21 44 24 51 28 60 26 56
Dixmaanb 29 66 33 73 26 60 27 67
Ex. Rosen 29 66 30 69 31 72 30 70
Cubic 14 39 17 46 21 51 16 44
Ex. Beal U63 10 27 10 27 10 27 10 27
Ex. TET 39 81 45 98 51 117 53 117
Gen.Tridiagonal-2 116 251 120 255 123 261 125 268
Diagonal 6 3 9 3 9 3 9 3 9
SUM 16 77 18 82 21 110 20 103
Total 592 1477 637 1542 653 1643 651 1600

Table 2: The percentage between PRP,LS,HS, and Gh for n = 100

Measurement PRP method LS method HS method Gh method

NOI 100% 99.69% 97.55% 90.66%
NOF 100% 97.38% 93.85% 89.9%



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1690

Table 3: Comparison between βHS , βPRP , βLS and βGh with n = 1000

Test functions βGh
k βHS

k βPRP
k βLSk

NOI NOF NOI NOF NOI NOF NOI NOF

Wood 32 74 36 84 35 81 37 87
Wolfe 58 108 59 119 58 116 59 118
Rosen 38 101 34 94 35 96 35 96
NON 30 76 33 84 39 94 32 83
Shallow 10 25 10 25 10 25 10 25
ENGVAL 1 20 44 22 46 23 46 22 46
Diagonal 2 54 207 62 225 58 214 59 219
Ex. BD1 21 44 23 48 24 50 23 48
Ex. Wood 27 61 30 67 30 67 29 65
Powell 39 110 41 109 56 129 52 121
Dixnaanc 5 15 5 15 5 15 5 15
DENSCHNF 27 62 29 67 31 73 30 69
Dixmaanb 35 75 39 89 32 67 34 72
Ex. Rosen 31 69 34 75 37 86 39 90
Cubic 16 41 18 50 21 59 20 55
Ex. Beal U63 12 29 12 29 12 29 12 29
Ex. TET 48 95 54 109 59 118 58 116
Gen.Tridiagonal-2 121 232 125 234 127 239 130 241
Diagonal 6 3 9 3 9 3 9 3 9
SUM 19 83 20 87 23 115 22 107
Total 647 1571 689 1663 722 1759 711 1691

Table 2 shows that the proposed βGh
k formula outperforms the classic βPRP

k , βLSk , and
βHS
k formulas in terms of percentage performance. We discovered that the proposed Gh al-

gorithm saves (NOI, 9.34%), (NOF, 10.1%), the LS algorithm saves (NOI, 0.31%), (NOF, 2.62%)
and the HS algorithm saves (NOI, 2.45%), (NOF, 6.15%), for n = 100.

Table 4 shows that the proposed βGh
k formula outperforms the classic βPRP

k , βLSk , and
βHS
k formulas in terms of percentage performance. We discovered that the proposed Gh al-

gorithm saves (NOI, 9.0%), (NOF, 7.1%), the LS algorithm saves (NOI, 1.52%), (NOF, 3.86%)
and the HS algorithm saves (NOI, 4.57%), (NOF, 5.46%), for n = 1000.

Figures (1) (a) and (b) depict the efficiency of the proposed method in terms of NOI
for n = 100 and 1000, respectively. Figures (2: (c) and (d)) demonstrate the effectiveness
of the suggested method in terms of NOF for n = 100 and 1000, respectively.

Table 4: The percentage between PRP,LS,HS, and Gh for n = 1000

Measurement PRP method LS method HS method Gh method

NOI 100% 98.48% 95.43% 91.0%
NOF 100% 96.14% 94.54% 92.9%



G. M. Al-Naemi / Eur. J. Pure Appl. Math, 15 (4) (2022), 1683-1693 1691

Figure 1: The comparison based (NOI) between βHS , βPRP , βLS , and new βGh (a) n=100 and (b) n=1000

Figure 2: The comparison based (NOI) between βHS , βPRP , βLS , and new βGh (c) n=100 and (d) n=1000

4. Conclusion

We present a new parameter βGh
k based on the memoryless self-scale DFP QN method

in this article. Any line search will suffice to ensure adequate descent property. We
also demonstrated that the Zoutendijk condition holds and that the method is globally
convergent by using some step-length technique. The numerical results demonstrated the
proposed algorithm’s efficiency when compared to some standard formulas.



REFERENCES 1692

Acknowledgements

The author wishes to thank the College of Comp. Sci. and Math. at the University of
Mosul for their encouragement.

References

[1] Abdelrahman A, Osman O O Yousif, and Khalid I Osman. A robust modification of
hestenes–stiefel conjugate gradient method with strong wolfe line search. Journal of
Mathematics and Statistics, 16(1):54–61, 2020.

[2] M Al-Baali. Descent property and global convergence of the fletcher—reeves method
with inexact line search. IMA Journal of Numerical Analysis, 5(1):121–124, 1985.

[3] G M Al-Naemi. A new modified hs algorithm with strong powell-wolfe line search for
unconstrained optimization. Eastern-European Journal of Enterprise Technologies,
2(4):116, 2022.

[4] N Andrei. An unconstrained optimization test functions collection. Advance Modeling
and Optimization, 10(1):147–161, 2008.

[5] N Andrei. Nonlinear conjugate gradient methods for unconstrained optimization.
Springer International Publishing, 2020.

[6] I Bongartz, A R Conn, N Gould, and Ph L Toint. Cute: Constrained and uncon-
strained testing environment. ACM Transactions on Mathematical Software (TOMS),
21(1):123–160, 1995.

[7] Y Chen. Global convergence of a new conjugate gradient method with wolfe type line
search. Journal of Information and Computing Science, 7(1):67–71, 2012.

[8] Dolan Elizabeth D and Moré Jorge J. Benchmarking optimization software with
performance profiles. Mathematical programming, 91(2):201–213, 2002.

[9] Y H Dai and Y Yuan. A nonlinear conjugate gradient method with a strong global
convergence property. SIAM Journal on optimization, 10(1):177–182, 1999.

[10] R Fletcher. Practical methods of optimization. John Wiley & Sons, 1987.

[11] R Fletcher and C M Reeves. Function minimization by conjugate gradients. The
computer journal, 7(2):149–154, 1964.

[12] M R Hestenes and E Stiefel. Methods of conjugate gradients for solving. Journal of
research of the National Bureau of Standards, 49(6):409, 1952.

[13] Ishak M I, Marjugi S M, and June L. A new modified conjugate gradient method
under the strong wolfe line search for solving unconstrained optimization problems.
Mathematical Modeling and Computing, 9(1):111–118, 2022.



REFERENCES 1693

[14] F N Jardow and G M Al-Naemi. A new hybrid conjugate gradient algorithm as a con-
vex combination of mmwu and rmil nonlinear problems. Journal of Interdisciplinary
Mathematics, 24(3):637–655, 2021.

[15] Y Liu and C Storey. Efficient generalized conjugate gradient algorithms, part 1:
theory. Journal of optimization theory and applications, 69(1):129–137, 1991.

[16] Oke M O, Oyelade T A, and Raji R A. On a modified conjugate gradient method for
solving nonlinear unconstrained optimization problems. European Journal of Basic
and Applied Sciences, 6(2):17–23, 2019.

[17] S S Oren and D G Luenberger. Self-scaling variable metric (ssvm) algorithms: Part
i: Criteria and sufficient conditions for scaling a class of algorithms. Management
Science, 20(5):845–862, 1974.

[18] E Polak and G Ribiere. Note sur la convergence de méthodes de directions conjuguées.
Revue française d’informatique et de recherche opérationnelle. Série rouge, 3(16):35–
43, 1969.

[19] B T Polyak. The conjugate gradient method in extremal problems. USSR Computa-
tional Mathematics and Mathematical Physics, 9(4):94–112, 1969.

[20] H Yabe and M Takano. Global convergence properties of new nonlinear conjugate
gradient methods for unconstrained optimization. Computational Optimization and
Application, 28:203–225, 2004.

[21] Salleh Zabidin, Alhamzi Ghaliah, Masmali Ibitsam, and Alhawarat Ahmad. A modi-
fied liu and storey conjugate gradient method for large scale unconstrained optimiza-
tion problems. Algorithms, 14(8):227, 2021.

[22] Yutao Zheng. A new family of dai-liao conjugate gradient methods with modi-
fied secant equation for unconstrained optimization. RAIRO-Operations Research,
55(6):3281–3291, 2021.

[23] G Zoutendijk. Nonlinear programming, computational methods. Integer and nonlin-
ear programming, pages 37–86, 1970.