EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS
Vol. 16, No. 2, 2023, 1059-1067
ISSN 1307-5543 – ejpam.com
Published by New York Business Global

A hybridization of the Hestenes-Stiefel and Dai-Yuan
Conjugate Gradient Methods

Yoksal A. Laylani1, Hisham M. Khudhur2,∗, Edrees M. Nori2, Khalil K. Abbo3

1 Department of Mathematics, College of Sciences, University of Kirkuk, Kirkuk, Iraq
2 Department of Mathematics, College of Computers Sciences and Mathematics,
University of Mosul, Mosul, Iraq
3 Department of Mathematics, College of Basic Education, University of Telafer, Tall’Afar,
Iraq

Abstract. The paper in discusses conjugate gradient methods, which are often used for uncon-
strained optimization and are the subject of this discussion. In the process of studying and imple-
menting conjugate gradient algorithms, it is standard practice to assume that the descent condition
is true. Despite the fact that this sort of approach very seldom results in search routes that slope
in a downward direction, this assumption is made routinely. As a result of this research, we pro-
pose a revised method known as the improved hybrid conjugate gradient technique. This method
is a convex combination of the Dai-Yuan and Hestenes-Stiefel methodologies. The descending
property and global convergence are both exhibited by the Wolfe line search. The numerical data
demonstrates that the strategy that was presented is an efficient one.

2020 Mathematics Subject Classifications: 90C30, 65K05, 49M37

Key Words and Phrases: Optimization, Conjugate gradient, Numerical results

1. Introduction

The following unconstrained optimization problems are dealt with in this paper:

minf(x), x ∈ Rn (1)

Where f : Rn → R is a continuously differentiable function that is bounded from below.
From an initial guess, a non linear conjugate gradient (CG) algorithm generates a sequence
of points {xk}, according to the formula for recurrence [6, 7]

xk+1 = xk + αkdk (2)

∗Corresponding author.
DOI: https://doi.org/10.29020/nybg.ejpam.v16i2.4746

Email addresses: yuksel@uokirkuk.edu.iq (Y. A. Laylani),
hisham892020@uomosul.edu.iq ( H. M. Khudhur), edreesnori@uomosul.edu.iq (E. M. Nori),
dr.khalil-khuder@uotelafer.edu.iq (K. K. Abbo)

https://www.ejpam.com 1059 © 2023 EJPAM All rights reserved.



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1060

where dk ∈ Rn search direction and αk ∈ R is a step length that is normally obtained
using the Wolfe method. [16, 19] and, the standard Wofle condition (SWC)

f (xk + αkdk)− f (xk) ≤ ραkg
T
k dk (3)

gTk+1dk ≥ σgTk dk (4)

with 0 < ρ < σ < 1, or by strong Wolfe condition(STWC)

f(xk + αkdk)− f(xk) ≤ ραkg
T
k dk (5)∣∣gTk+1dk

∣∣ ≤ −σgTk dk, (6)

and compute the search direction from the following equation

dk+1 = −gk+1 + βksk, d0 = −g0 (7)

Here βk the conjugate gradient parameter which is a scalar known as gk = ∇f(xk)
and sk = xk+1 − xk. Different conjugate gradient algorithms corresponding to various
parameter choices βk see [3, 11, 13, 17, 18, 20, 23]. As a consequence, every conjugate
gradient algorithm formula definition would be the same is crucial βk The formulas are
well-known. βk are as follows;

βFR
k =

gTk+1gk+1

gTk gk
, βDY

k =
∥gk+1∥2

yTk sk
, βCD

k =
gTk+1gk+1

−dTk gk

βPR
k =

yTk gk+1

gTk gk
, βHS

k =
yTk gk+1

yTk dk
, βLS

k =
yTk gk+1

−dTk gk

FR denotes Fletcher and Reeves [10] , HS denotes Hestenes and Steifel [13], PR denotes
Polak and Ribiere [19] , DY denotes Dai and Yuan [8] and LS denotes Liu and Storey.
It’s worth noting that these formulas for βk they are similar when the objective function
is a strictly convex quadratic function, and αk is an exact one-dimensional minimizer [9].
The following is the outline of the document. Section 2 introduces and demonstrates our
proposed method (HKYE) for generating descent directions. Its convergence analysis is
seen in section 3. In section 4, you’ll find some numerical experiments.

2. Hybrid conjugate gradient method

The traditional numerator conjugate gradient methods ∥gk+1∥2 in relation to the up-
date parameter βk (FR, DY, CD) Although they have good convergence properties, their
functional performance is limited. While the methods with numerator gTk+1yksuch as (PR,
HS, LS)often have better computation performance, but they may not generally be con-
vergent. In this section we try to introduce a new hybrid conjugate gradient method as
follows: Consider the following search direction defined by

dk+1 = − gk+1 + [(1− θk)β
HS
k + θβDY

k ]sk (8)



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1061

By using the conjugacy condition (Dai-Liao)

dTk+1yk = −tgTk+1sk

and let

t =
(sTk gk+1)

2

(yTk sk)
2

We get the equation (9)

θk =
(sTk gk+1)

3

(yTk sk)
2

(9)

Note that θk < 1, since sTk gk+1 < yTk sk, but it may be θk < 0, to avoid this situation we
take θ = 0. Therefor, equation (8) becomes

dk+1 = − gk+1 + [βHS
k ]sk (10)

We call the method defined by equation (8)(HKYE) method.

2.1. New Algorithms

Step 1. Given x1 ∈ Rn, ε > 0, d1 = −g1.

Step 2. If ∥gk∥2 < ε then stop. else go to Step3.

Step 3. Compute an αk > 0satisfying (5)and (6)

Step 4. Let xk+1 = xk + αkdk. If ∥gk+1∥2 < ε then stop, else go to Step5

Step 5 Compute θand βk. If θ > 1or θ < 0then compute dk+1 = −gk+1 + βHS
k sk else

dk+1 = −gk+1 +
{(

1− θβHS
k + θβDY

k

}
sk

2.2. Descent property and global convergence analysis

Next we will show that our CG method (8) satisfies the descent property and global
converges.

Theorem 1. Let {θk} and {dk+1} be the sequences generated by Eq. (8) and (9) with
strong wolfe conditions and assume that gTk+1gk ≥ 0 and σ ≤ 1

2 . Then the search directions

dk+1 satisfies sufficient descent condition gTk+1dk ≤ −c ∥gk∥ Where c is constant.

Proof. The prove is by in induction that is if k = 0 then d1 = −g1 and gT1 d1 = −∥g1∥
Assume that gTk dk ≤ −c ∥gk∥ to prove for k = k + 1 we have yTk sk = gTk+1sk − gTk sk ≥
(σ − 1) gTk dk > 0 and θk =

(sTk gk+1)
3

(yTk sk)2
<1, if θk < 0. We set θk = 0 therefor we assume

0 < θk < 1. Then

dk+1 = −gk+1 +

[
(1− θk)

yTk gk+1

yTk sk
− θk

gTk+1gk+1

yTk sk

]
sk



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1062

Multiply both sides by gTk+1

gTk+1dk+1 = −gTk+1gk+1 +

[
(1− θk)

yTk gk+1

yTk sk
− θk

gTk+1gk+1

yTk sk

]
gTk+1sk

≤ −gTk+1gk+1 +

[
(1− θk)

yTk gk+1g
T
k+1sk

(σ − 1) gTk dk
−

(
gTk+1sk

)4
(yTk sk ∗ yTk sk)

gTk+1gk+1

yTk sk

]
Since the last term is positive

∴ gTk+1dk+1 ≤ −gTk+1gk+1+

[
(1− θk)

∣∣yTk gk+1

∣∣ ∣∣gTk+1sk
∣∣

(σ − 1) gTk dk

]
≤ −gTk+1gk+1+

[
(1− θk)

∣∣yTk gk+1

∣∣ (−σgTk dk)

(σ − 1) gTk dk

]

∴ gTk+1dk+1 ≤ −gTk+1gk+1+

[
(1− θk)

∣∣yTk gk+1

∣∣ (−σ)

(σ − 1)

]
= −gTk+1gk+1+

[
(1− θk)

ó
∣∣yTk gk+1

∣∣
(1− σ)

]
Since gTk+1gk ≥ 0 and

∣∣yTk gk+1

∣∣ = ∣∣gTk+1 (|gk+1 − gk|)
∣∣ ≤ ||gk+1||2

∴ gTk+1dk+1 ≤ −gTk+1gk+1 +

[
(1− θk)

σgTk+1gk+1

(1− σ)

]
=

(
−1 +

[
σ (1− θk)

(1− σ)

])
gTk+1gk+1

dTk+1gk+1 ≤ −c ∥gk+1∥2 where c = (1− σ(1−θ)
(1−σ) ), the prove is complete.

The following assumptions are used to demonstrate the proposed algorithm’s global
convergence.

Assumption 1. 1. In the level set, the objective function f(x) is bounded below

Ω = {x ∈ Rn : f(x) ≤ f(x1)} ;

2. In some parts of town N ofΩ,f There exists a constant that is constantly differentiable
and has a Lipschitz continuous gradient L > 0 such that:

∥g(x)− g(y)∥ ≤ L ∥x− y∥ , ∀x, y ∈ N (11)

It’s worth noting that these assumptions suggest the existence of a constant Γ, implying
that ∥gk∥ ≤ Γ, for any x in the level set Ω, and ∥x∥ ≤ B, ∀x ∈ Ω

Lemma 1. [1, 2, 12, 14, 15] Assume that dkis a descent direction and gk satisfies the
Lipschitz condition∥g(x)− g(y)∥ ≤ L ∥x− y∥for all xon the linking section of a line xk
and xk+1 L is the constant in this equation. If Wolfe’s condition is met, the line quest is
successful then

αk ≥ 1− σ

L

∣∣gTk dk∣∣
∥dk∥2



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1063

Lemma 2. [4, 5, 8, 15, 21, 22] Assume the Assumption is right. The descent condition
is satisfied if the conjugate gradient method is used.

∞∑
k=1

(gTk dk)
2

∥dk∥2
< +∞ (12)

Theorem 2. Assume the Assumption 2.3. is true. Let {gk}and {dk} be the sequence
generated by the algorithm (HKYE) when subjected to a strong Wolfe line search condition.
Then lim inf ∥gk∥

k→∞
= 0

Proof. It follows from Assumption 2.3. that there is a positive constant Γ >0 Such

that ∥g(x)∥ ≤ Γ for all x ∈ Ω, and Theorem 2.1. αk ≥ (1−σ)
L

(gTk dk)
2

∥dk∥2
= λ If the theorem

is incorrect, then a positive constant must exist >0 such that is true for all k sufficiently
large

∥gk∥ ≥ µ (13)

By the Eq(6)and Theorem 2.1., it results that

yTk sk = gTk+1sk − gTk sk ≥ −(1− σ)gTk sk ≥ c(1− σ)gk
2 (14)

Lipschitz continuity of the gradient, gives

yk = gi+1 − gk ≤ L xk+1 − xk ≤ LR (15)

Where R =max{∥x− y∥ ;x, y ∈ Ω } is diameter of the level. Now

|βk| =
∣∣(1− θk)β

HS
k + θβDY

k

∣∣
≤

∣∣βHS
k

∣∣+ ∣∣βDY
k

∣∣
=

∣∣αkg
T
k yk

∣∣∣∣yTk sk∣∣ +

∣∣αkg
T
k+1gk+1

∣∣∣∣yTk sk∣∣
≤ αk ∥gk+1∥ ∥yk∥

c ∥gk∥
+

αk ∥gk+1∥
c(1− σ) ∥gk∥

≤ αkΓLR

c(1− σ)
+

αkΓ
2

c(1− σ)µ2
= V

Therefor
∥dk+1∥ = ∥gk+1∥+ |βk| ∥xk+1 − xk∥

= ∥gk+1∥+ |βk|∥sk∥
αk

≤ Γ + V R
µ ≡ Q

(16)

Which gives ∑
k≥0

1

∥dk∥
= ∞ (17)



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1064

Theorem 2.1. and Eq(13) and the Zoutendijk condition, on the other hand, result from
adequate descent

c2Γ4
∑
k≥0

1

∥dk∥
≤

∑
k≥0

c2 ∥gk∥4

∥dk∥2
≤

∑
k≥0

(gTk dk)
2

∥dk∥2
= ∞

As a result of the inconsistency with (17), (13) does not hold, and as a result

lim inf ∥gk∥
k→∞

= 0

The prove is complete.

3. Comparisons and numerical results

The computational output of a Fortran implementation of the latest suggested (HKYE)
algorithm on a set of 400 unconstrained optimization test problems is presented in this
section. The unconstrained optimization problems are the test problems in [6]. We chose
40 test problems that were either extended or generalized. Each problem is evaluated ten
times, with the number of variables increasingly increasing n=100,200,. . . ,1000. The code
for Fortran from We compare our results to those of the HS and DY algorithms, as well
as the Dolan and More performance profiles [23]. The strong Wolfe line search conditions
are used in all algorithms withρ = 0.0001andσ = 0.4. The same criteria for stopping
∥gk∥2 < 10−6is used. Algorithm comparisons are presented in the following context. Let
fALG1
i and fAIG2

i to be the best value found by ALG1 andALG2 for problem i=1,..,400,
respectively. If the performance of ALG1 was better than the performance of ALG2 in
the specific problem I we can assume that the performance of ALG1 was better than the
performance of ALG2. ∣∣fALG1

i − fAIG2
i

∣∣ < 10−3

The number of iterations (#ite), function-gradient evaluations (#fg), or CPU time corre-
sponding to ALG1 was less than the number of (#ite), (#fg), or CPU time corresponding
to ALG2, respectively. To compare the performance of (HKYE) to that of HS and DY, we
consider the number of iterations (#ite) in fig.1. Figure 2 shows how these algorithms per-
form when evaluating the number of function gradients (#fg), while Figure 3 shows how
they perform when evaluating CPU time. We can see that (HKYE) is the best performer.
These codes vary in their search path since they use the same strong Wolfe line search and
stopping criteria. As a result, it appears that (HKYE) generates the best search direction
among these methods.



H. M. Khudhur et al. / Eur. J. Pure Appl. Math, 16 (2) (2023), 1059-1067 1065

Figure 1: Iteration-based efficiency

Figure 2: Performance by function

Figure 3: Time-Based Performance



REFERENCES 1066

4. Conclusion

We presented in this research a new type of conjugate gradient technique for solving
unconstrained optimization problems, and the proposed algorithm has shown high effi-
ciency in solving these problems with the least number of iterations and higher accuracy
in reaching the approximate solution of the function.

References

[1] K K Abbo and H M Khudhur. New a hybrid conjugate gradient fletcher-reeves
and polak-ribiere algorithm for unconstrained optimization. Tikrit Journal of Pure
Science, 21:124–129, 2016.

[2] K K Abbo and H M Khudhur. New a hybrid hestenes-stiefel and dai-yuan conjugate
gradient algorithms for unconstrained optimization. Tikrit Journal of Pure Science,
21:118–123, 2016.

[3] K K Abbo, Y A Laylani, and H M Khudhur. Proposed new scaled conjugate gradient
algorithm for unconstrained optimization. Int. J. Enhanc. Res. Sci. Technol. Eng,
5:269–276, 2016.

[4] K K Abbo, Y A Laylani, and H M Khudhur. A new spectral conjugate gradient
algorithm for unconstrained optimization. Int. J. Math. Comput. Appl. Res, 8:1–9,
2018.

[5] Z M Abdullah, M Hameed, H M Khudhur, and M A Khaleel. Modified new conjugate
gradient method for unconstrained optimization. Tikrit Journal of Pure Science,
24:86–90, 2019.

[6] N Andrei. An unconstrained optimization test functions collection. Adv. Model.
Optim, 10:147–161, 2008.

[7] Y Dai and L Liao. New conjugacy conditions and related non-linear cg methods’
appl. Math optim.,(43). Spring-verlag, New York, 2001.

[8] Y H Dai and Y Yuan. A nonlinear conjugate gradient method with a strong global
convergence property. SIAM Journal on optimization, 10:177–182, 1999.

[9] R Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.

[10] R Fletcher and C M Reeves. Function minimization by conjugate gradients. The
computer journal, 7:149–154, 1964.

[11] J C Gilbert and J Nocedal. Global convergence properties of conjugate gradient
methods for optimization. SIAM Journal on optimization, 2:21–42, 1992.

[12] W Hager and H Zhang. A survey of nonlinear conjugate gradient methods. Pacific
journal of Optimization, 2:35–58, 2006.



REFERENCES 1067

[13] M R Hestenes and E Stiefel. Methods of conjugate gradients for solving linear systems.
Journal of research of the National Bureau of Standards, 49:409–436, 1952.

[14] H M Khudhur and K K Abbo. A new conjugate gradient method for learning fuzzy
neural networks. Journal of Multidisciplinary Modeling and Optimization, 3:57–69,
2020.

[15] H M Khudhur and K K Abbo. A new type of conjugate gradient technique for solving
fuzzy nonlinear algebraic equations. In Journal of Physics: Conference Series, volume
1879, page 022111, 2021.

[16] Y A Laylani, B A Hassan, and H M Khudhur. A new class of optimization meth-
ods based on coefficient conjugate gradient. European Journal of Pure and Applied
Mathematics, 15:1908–1916, 2022.

[17] J Liu and S Wang. Modified nonlinear conjugate gradient method with sufficient
descent condition for unconstrained optimization. Journal of Inequalities and Appli-
cations, 2011:1–12, 2011.

[18] J Nocedal and S J Wright. Numerical optimization. 2006.

[19] E Polak and G Ribiere. Note sur la convergence de méthodes de directions conjuguées.
Revue française d’informatique et de recherche opérationnelle. Série rouge, 3:35–43,
1969.

[20] M Powell. A survey of numerical methods for unconstrained optimization. SIAM
review, 12:79–97, 1970.

[21] A Sahiner, I A M Abdulhamid, and S A Ibrahem. A new filled function method with
two parameters in a directional search. Journal of Multidisciplinary Modeling and
Optimization, 2:34–42, 2019.

[22] A Sahiner and S A Ibrahem. A new global optimization technique by auxiliary
function method in a directional search. Optimization Letters, 13:309–323, 2019.

[23] P Wolfe. Convergence conditions for ascent methods. ii: Some corrections. SIAM
review, 13:185–188, 1971.