EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS
Vol. 14, No. 4, 2021, 1429-1456
ISSN 1307-5543 – ejpam.com
Published by New York Business Global

A Descent Four-Term of Liu and Storey Conjugate
Gradient Method for Large Scale Unconstrained

Optimization Problems

Ahmad Alhawarat1, Hanan Alolaiyan2, Ibtisam A. Masmali3, Zabidin Salleh4,
Shahrina Ismail5,∗

1 Division of Computational Mathematics and Engineering, Institute for Computational
Science, Ton Duc Thang University, Ho Chi Minh City, Vietnam
2 Department of Mathematics, College of Science, King Saud University, Riyadh 11451,
Saudi Arabia
3 Department of Mathematics, College of Science, Jazan University, Jazan, Saudi Arabia
4 Department of Mathematics, Faculty of Ocean Engineering Technology and Informatics,
Universiti Malaysia Terengganu, Terengganu, Malaysia
5 Financial Mathematics Program, Faculty of Science and Technology, Universiti Sains
Islam Malaysia, Bandar Baru Nilai, 71800, Nilai, Negeri Sembilan, Malaysia

Abstract. The conjugate gradient (CG) method is a useful tool for obtaining the optimum
point for unconstrained optimization problems since it does not require a second derivative or its
approximations. Moreover, the conjugate gradient method can be applied in many fields such as
machine learning, deep learning, neural network, and many others. This paper constructs a four-
term conjugate gradient method that satisfies the descent property and convergence properties
to obtain the stationary point. The new modification was constructed based on Liu and Storey’s
conjugate gradient method, two-term conjugate gradient method, and three-term conjugate gra-
dient method. To analyze the efficiency and robustness, we used more than 150 optimization
functions from the CUTEst library with different dimensions and shapes. The numerical re-
sults show that the new modification outperforms the recent conjugate gradient methods such
as CG-Descent, Dai and Liao, and others in terms of number of functions evaluations, number
of gradient evaluations, number of iterations, and CPU time.

2020 Mathematics Subject Classifications: 65K10, 90C25, 90C26

Key Words and Phrases: Conjugate gradient method, Wolfe Powell line search, Descent
condition, Convergence

∗Corresponding author.
DOI: https://doi.org/10.29020/nybg.ejpam.v14i4.4128

Email addresses: shahrinaismail@usim.edu.my (S. Ismail)

http://www.ejpam.com 1429 © 2021 EJPAM All rights reserved.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1430

1. Introduction

To solve unconstrained optimization problems, normally, we use the conjugate gradient
(CG) method since it does not require memory storage or the second derivative of the
objective function. We consider the following problem:

min f(x), x ∈ Rn, (1)

where f(x)satisfies the following assumption.

Assumption 1
A. f : Rn → R is a continuous and differentiable function, and the gradient is available.
B. The level set Ψ = {x |f(x) ≤ f(x1)} is bounded, that is, a positive constant ϑ exists
such that

∥x∥ ≤ ϑ, ∀x ∈ Ψ.

C. In some neighbourhood Q of Ψ, f is continuously differentiable, and its gradient
is Lipschitz continuous; that is, for all, x, y ∈ Q, there exists a constant L > 0 such that

∥g(x)− g(y)∥ ≤ L ∥x− y∥

In addition, from Assumption 1, we can conclude that there exists a positive constant
B such that

∥g(u)∥ ≤ B, ∀u ∈ N.

The CG method generates a sequence of xk starting from the initial point x1 by the
equation

xk+1 = xk + αkdk, k = 1, 2, ..., (2)

where xk+1is the next iteration.
The search direction dk in the CG method is defined by the following equation

dk =

{
−gk,
−gk + βkdk−1,

if k = 1,
if k ≥ 2,

(3)

where gk = g(xk) = ∇f and βk is known as the CG formula. Note that for k = 1,we
use the steepest descent method. To obtain the steplength (αk), we have the following
two-line searches:

A - Exact line search: To find the step size such that the objective function in
the search direction is minimized i.e.

f(xk + αkdk) = min f(xk + αdk), α ≥ 0.

However, exact optimal step size generally can not be found, and it is expensive to find
almost exact step size [29]. Thus, we use inexact line search, as discussed in part B.

B - Inexact line search



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1431

To avoid expensive computational to obtain the step size by exact line search, we use
inexact line search. Here, the most popular inexact line search is Wolfe Powell (WP)
line search, which is divided into two parts:
B1- The first part is weak Wolfe Powell (WWP) [30, 31] and is given by the following
equations:

f(xk + αkdk) ≤ f(xk) + δαkg
T
k dk, g(xk + αkdk)

Tdk ≥ σ gTk dk. (4)

B2- The second part is strong Wolfe Powell (SWP) line search, which is defined by
equation (4) and

|g(xk + αkdk)
Tdk| ≤ σ

∣∣gTk dk∣∣ , (5)

where 0 < δ < σ < 1.
The most well known classical CG formulas are Hestenses–Stiefel (HS)[19], Polak–
Ribiere–Polyak (PRP)[25], Liu and Storey (LS)[23], Fletcher–Reeves(FR)[14], Fletcher
(CD)[13], Dai and Yuan (DY)[11], and these formulas are given as follows:

βHS
k =

gTk yk−1

dTk−1yk−1
, βPRP

k =
gTk yk−1

∥gk−1∥2
, βLS

k = −
gTk yk−1

dTk−1gk−1
,

βFR
k =

∥gk∥2

∥gk−1∥2
, βCD

k = − ∥gk∥2

dTk−1gk−1
, βDY

k =
∥gk∥2

dTk−1gk−1
,

where yk−1 = gk − gk−1.
The global convergence properties of the FR method were studied by Zoutendijk [33] and
Al-Baali [5]. The global convergence of the PRP method for convex objective function
under exact line search was proved by Polak and Ribere in [25]. Later, Powell [26]
gave out a counterexample showing that there exists a non-convex function, where PRP
and HS CG methods can cycle infinitely without getting a solution. Therefore, Powell
suggested the importance of achieving the global convergence of PRP and HS methods,
which should be non-negative. Meanwhile, Gilbert and Nocedal [15] proved that non-
negative PRP, i.e. βk = max{βPRP

k , 0 }, is globally convergent under complicated line
searches. Alhawarat et al. [2] also proposed the following non-negative CG formula with
new restart property as follows

βAZPRP
k =

{
∥gk∥2−µk|gTk gk−1|

∥gk−1∥2
, if ∥gk∥2 > µk

∣∣gTk gk−1

∣∣ ,
0, otherwise,

where ∥·∥ represents the Euclidean norm, while µk is defined as follows:

µk =
∥xk − xk−1∥

∥yk−1∥
.

Furthermore, Dai and Liao [10] proposed the following CG formula with a new conjugacy
condition as follows:



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1432

βDL
k =

gTk yk−1

dTk−1yk−1
− t

gTk sk−1

dTk−1yk−1
= βHS

k − t
gTk sk−1

dTk−1yk−1
. (6)

In addition, Hager and Zhang [17, 18] presented the following CG formula based on
equation (6) given by

βHZ
k = max {βN

k , ηk}, (7)

where βN
k = 1

dTk yk
(yk − 2dk

∥yk∥2

dTk yk
)T gk, ηk = − 1

∥dk∥ min{η, ∥gk∥} , and η > 0 is a constant.

Note that if t = 2∥yk∥2

sTk yk
, then βN

k = βDY
k .

Alhawarat et al. [1] also presented a four-term CG method based on equation (6) as
follows:

dFTCGHS
k = −gk +

(
βHS
k − tk

gTk sk−1

yTk−1dk−1

)
dk−1 −

(
gTk dk−1

yTk−1dk−1

)
(yk−1 + sk−1). (8)

Furthermore, Zabidin et al. [32] presented the following CG formula based on [11] as
follows

βLS+
k =

 −∥gk∥2−µk|gTk gk−1|
dTk−1gk−1

if ∥gk∥2 > µk

∣∣gTk gk−1

∣∣ ,
βDL−HS
k otherwise,

(9)

where ∥·∥ represents the Euclidean norm and

βDL−HS
k = − µk

gTk sk−1

dTk−1yk−1
.

Moreover, Liu et al. [22] proposed the three-term CG method as follows

dk = −gk +

(
βLS
k −

∥gk−1∥2 gTk dk−1

(dTk−1gk−1)2

)
dk−1 +

(
gTk dk−1

dTk−1yk−1

)
gk−1,

with the following assumption (
gTk dk−1

dTk−1gk−1

)
> ν ∈ (0, 1).

Additionally, Yao et al. [27] proposed three terms of CG with a new choice of t as
follows:

dk+1 = −gk+1 +

(
gTk yk − tkg

T
k+1sk

yTk dk

)
dk +

gTk+1dk

yTk dk
yk.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1433

Based on the SWP line search, Yao et al. [27] selected tk to satisfy the descent condition
as follows:

tk >
∥yk∥2

yTk sk
.

The CG method can be applied in many fields such as medical science, neural network,
image restoration, machine learning, finance, economics and many others. For example,
Alhawarat et al. [1] presented an application for image restoration using the CG method
to restore true images from damaged images with an efficient number of iterations and
CPU time. Moreover, we can test the quality using the root-mean-square error (rmse)
between the original image and the restored image as follows:

rmse =
∥ς − ιk∥2

∥ς∥
.

Here, ςand ιk are the true image and restored images. For more about the CG method
and its applications, the reader can refer to the following references [3, 4, 6, 20].

2. Preliminary

Definition 2.1 [8]. A sequence of real numbers is a function whose domain is the set
of natural numbers N = {1, 2, ...} and whose range is contained in R. A sequence an is
considered increasing if a1 < a2 < a3 < ... < ak... that is, ak < ak+1 for all k. Similarly,
the decreasing sequences can be defined.

Definition 2.2 [8]. A sequence an is said to converge to the limit L for any given ε > 0.
Then, there is a positive integer N such that |an − L| < ε for all n ≥ N. In this case,
we have lim

n→∞
an = L. A sequence that does not converge to some finite limit is called

to diverge.

Definition 2.3 [16]. An infinite series is an expression that can be written in the form
of

∞∑
k=1

uk = u1 + u2 + u3 + ...+ uk + ....

The number u1, u2, u3, ... is called the term of the series.

Definition 2.4 [16]. Let an be the sequence of partial sums of the series u1+u2+u3+
...+ uk + .... If the sequence an converges to a limitA, then the series is convergent to A
and A is called the sum of the series. This is defined by:

A =

∞∑
k=1

uk.

If the sequence of partial sums diverges, then the series is diverging.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1434

Theorem 2.1 [16]. A geometric series
∑∞

k=1 ar
k = a+ ar+ ar2 + ...+ ark + ... , where

(a ̸= 0). Then, it is convergent if |r| ≤ 1and diverges if |r| > 1. If the series converges,
then the sum is,

∑∞
k=1 ar

k = a
1−r .

Definition 2.5 [8] For f :Rn → R a function that is continuous and differentiable, then
there exists at any point x ∈ Rn, a vector of first-order partial derivatives or a gradient
vector given by

∇f(x) =


∂f(x)
∂x1
∂f(x)
∂x2
...

∂f(x)
∂xn

 = g(x).

Definition 2.6 [28]. Let function f :Rn → R be twice continuously differentiable. Then,
at a point x ∈ Rn, there exists a matrix of second-order partial derivatives or a Hessian
matrix given by

∇2f(x) = H(x) =


∂2f(x)
∂x1

2
∂2f(x)
∂x1∂x2

· · · ∂2f(x)
∂x1∂xm

∂2f(x)
∂x2∂x1

∂2f(x)
∂x2

2

∂2f(x)
∂x2∂xm

...
...

...
∂2f(x)
∂xn∂x1

∂2f(x)
∂xn∂2

· · · ∂2f(x)
∂xn∂xm

 ,

where m, n ∈ N.
Definition 2.7 [8]. Quadratic form of a function f :Rn → R is denoted by f(x) =
1
2x

TQx− bx, where x ∈Rn, Q is n× n real matrix, and b is a constant.

Definition 2.8 [7]. A set Cis convex if the line segment between any points in C lies
in C. For any x1, x2 ∈ Cand any θ with 0 ≤ θ ≤ 1, the θ x1 + (1− θ)x2 ∈ C.

Definition 2.9 [7]. A function f :Rn → R is convex if the domain of f is a convex set
for all x, y ∈ domain f , and

f(θ x+ (1− θ)y) ≤ θ f(x) + (1− θ)f(y),

where 0 ≤ θ ≤ 1.

3. The new search direction and its motivation

The CG method has become very rich in recent years. The main goal is to develop
a new CG method robust and efficient to solve large scale unconstrained optimization
problems. In addition, the CG method can be applied in several fields, as mentioned
before. Thus, to overcome the convergence properties of the LS CG method and to
improve the efficiency of dFTCGHS

k , we construct the following search direction based on
DL and dFTCGHS

k search directions as follows



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1435

dFTCGLS
k = −gk+

(
−

gTk yk−1

dTk−1gk−1
− tk

gTk sk−1

dTk−1gk−1

)
dk−1+

(
gTk dk−1

dTk−1gk−1

)
(yk−1−sk−1), (10)

where tk = ∥sk∥
∥yk−1∥ . In following sections, we assume that:

χk = βLS
k − tk

gTk sk−1

dTk−1gk−1
and θk =

(
gTk dk−1

dTk−1gk−1

)
.

If we use exact line search, the equation can be reduced to the original LS. Since

gTk dk−1 = 0,

and
gTk sk−1 = αkg

T
k dk−1,

we obtain
dFTCGHS
k = −gk +

(
βLS
k

)
dk−1.

Algorithm 1 describes the steps of CG method to obtain the stationary point using SWP
line search and equation (12) with stopping criteria∥gk∥ ≤ 10−6.
Algorithm 1
Step 1. Set a starting point x1. This initial point can be arbitrary or standard for
scientific functions. The initial search direction is the negative gradient, i.e. d1 = −g1.
Let k := 1.
Step 2. If the stopping condition is satisfied, then stop.
Step 3. Compute the search direction dk based on equation (2) using equation (10).
Step 4. Compute the step size αk using equations (4) and (5).
Step 5. Update xk+1 based on equation (2).
Step 6. Set k := k + 1 and go to Step 2.

4. Convergence analysis of the CG method with Algorithm 1

The descent condition (downhill condition) is given by the following equation

gTk dk < 0, ∀k ≥ 1, (11)

which is useful in the study of the CG method and serves important rule in the proof of
convergence analysis. Al-baali [5] modified equation (11) to the following form and used
it to prove the FR method given by

gTk dk ≤ −c ∥gk∥2 , ∀k ≥ 1, (12)

where c ∈ (0, 1).



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1436

4.1. The descent property of the new search direction

In the next theorem, we show that the search direction in equation (10) ensures that the
sufficient descent condition (12) is satisfied with the SWP line search.

Theorem 4.1 Let the sequences {xk} and {dk} be generated using equations in Al-
gorithm 1, where αk is computed using SWP line search. Then, the sufficient descent
condition holds.

Proof. Multiply (12) by gTk , which yields

gTk dk = −∥gk∥2 +

(
−

gTk yk−1

dTk−1gk−1
− t

gTk sk−1

dTk−1gk−1

)
gTk dk−1 +

(
gTk dk−1

dTk−1gk−1

)
gTk (yk−1 − sk−1),

= −∥gk∥2+

(
−

gTk yk−1

dTk−1gk−1

)
gTk dk−1−t

(
gTk sk−1

dTk−1gk−1

)
gTk dk−1+

(
gTk dk−1

dTk−1gk−1

)
gTk yk−1−

(
gTk dk−1

dTk−1gk−1

)
gTk sk−1,

= −∥gk∥2 − tαk

(
gTk dk−1

dTk−1yk−1

)
gTk dk−1 − αk

(
gTk dk−1

dTk−1yk−1

)
gTk dk−1,

= −∥gk∥2 − αk

(∥∥gTk dk−1

∥∥2
dTk−1yk−1

)
(t+ 1) .

From the SWP line search, we obtain

dTk−1yk−1 > 0.

Thus, gTk dk ≤ −∥gk∥2 . The proof is complete. ■

For example, the Zoutendijk condition [33] presented a useful Lemma for analyzing the
convergence property of the CG method. The Lemma is given as follows:

Lemma 4.1 Let Assumption 1 holds. Consider any CG method in the form (2) and
(5), where αk satisfies the WWP line search, in which the search direction satisfies the
descent condition. Then, the following condition holds:

∞∑
k=0

(gTk dk)
2

∥dk∥2
< ∞. (13)

4.2. Convergence of Algorithm 1 with convex functions

Dai and Liao [9] proposed useful theorems (Theorem 3.2, Theorem 3.3, and Corollary
3.1) for convergence analysis of the CG method as follows:



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1437

Theorem 4.2. Suppose Assumption 1 holds. Consider any method in the form (2)
and (5), where the search directed is descent with the step length obtained by SWP line
search. Then, either

lim inf
k→∞

∥gk∥ = 0, (14)

or

∞∑
k=0

∥gk∥4

∥dk∥2
< ∞. (15)

Theorem 4.3. Suppose Assumption 1 holds. Consider any method in the form (2) and
(5), where the search directed is descent and the step length is obtained by SWP line
search. If

∞∑
k=0

∥gk∥t

∥dk∥2
= +∞. (16)

Then, for any t ∈ [0, 4], the method converges in the sense that lim inf
k→∞

∥gk∥ = 0.

Proof. Suppose that
lim inf
k→∞

∥gk∥ ̸= 0.

Then, from Theorem 4.2, we obtain

∞∑
k=0

∥gk∥4

∥dk∥2
< ∞, (17)

because ∥gk∥ is bounded away from zero andt ∈ [0, 4]. It is easy to see that (17)
contradicts (16). Thus, the theorem is true and the proof is complete. ■

Corollary 4.1. Suppose that Assumption 1 holds. Consider any conjugate gradient
method in the form of equations (2) and (5), where dk is a descent direction and αk is
obtained by the strong Wolfe line search. If

∞∑
k≥1

1

∥dk∥2
= ∞, (18)

then
lim inf
k→∞

∥gk∥ = 0.

Proof. From Theorem 4.3 and using t = 0, we obtain lim inf
k→∞

∥gk∥ = 0.

The following theorem shows that the new search direction satisfies the convergences
analysis with convex functions.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1438

Theorem 4.4. Suppose that Assumption 1 holds. Consider the CG method in the
forms of equations (2) and (10), and dk as a descent direction by using Theorem 3.1,
where αk is obtained using strong Wolfe-Powell line search. It f(x) is a uniformly convex
function, then lim inf

k→∞
∥gk∥ = 0.

Proof. Because the function f(x)is uniformly convex, there exists a positive constantϖsuch
that

ϖ ∥x− y∥2 ≤ (∇f(x)−∇f(y))T (x− y).

For all x, y ∈ Ψ, we have

dk−1yk−1 ≥ ϖαk−1 ∥dk−1∥2 . (19)

Using equation (14) and triangular inequality, we get

∥dk∥ ≤ ∥gk∥+

( ∣∣gTk yk−1

∣∣∣∣dTk−1gk−1

∣∣ + tk

∣∣gTk sk−1

∣∣∣∣dTk−1gk−1

∣∣
)
∥dk−1∥+

( ∣∣gTk dk−1

∣∣∣∣dTk−1gk−1

∣∣
)
(∥yk−1∥+∥sk−1∥).

By using equation (18), we have

∥dk∥ ≤ ∥gk∥+

(
(σ + 1)

∣∣gTk yk−1

∣∣
ϖαk−1 ∥dk−1∥2

+ tk
(σ + 1)

∣∣gTk sk−1

∣∣
ϖαk−1 ∥dk−1∥2

)
∥dk−1∥+

(
(σ + 1)

∣∣gTk dk−1

∣∣
ϖαk−1 ∥dk−1∥2

)
(∥yk−1∥+∥sk−1∥).

Also, using triangular inequality and Assumption 1 yields

∥dk∥ ≤ ∥gk∥+
(
Lαk−1(σ+1)∥gk∥∥dk−1∥2

ϖαk−1∥dk−1∥2
+ t

αk−1(σ+1)∥gk∥∥dk−1∥2

ϖαk−1∥dk−1∥2

)
+
(
(σ+1)∥gk∥∥dk−1∥2

ϖαk−1∥dk−1∥2

)
(Lαk−1+

αk−1),

≤ ∥gk∥+ (σ + 1)
(
L∥gk∥
ϖ + tk

∥gk∥
ϖ

)
+ (σ + 1)

(
∥gk∥
ϖ

)
(L+ 1),

≤ ∥gk∥+ (σ + 1)∥gk∥ϖ (2L+ tk + 1).

By using Assumption 1, we obtain

∥dk∥ ≤ B +
B

ϖ
(σ + 1)(2L+ tk + 1).

Let

B +
B

ϖ
(2L+ tk + 1)(σ + 1) = M,

where M is constant; thus
∥dk∥ ≤ M,



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1439

which implies the truth of equation (18). Thus, using Corollary 3.1, we have

lim inf
k→∞

∥gk∥ = 0.

The proof is now complete. ■

4.3. Convergence of Algorithm 1 with general nonlinear functions

The following restriction for χk is essential to establish the convergence analysis for the
new search direction. The main importance of this restriction is to avoid the CG method
multiplayer being non-negative

χ+
k
= max

{
0, βLS

k − t
gTk sk−1

dTk−1yk−1

}
.

Thus, equation (10) becomes as follows

dk = −gk + χ+
k dk−1 + θk(yk−1 − sk−1).

Lemma 4.2. Assume that Assumption 1 holds and the sequences {gk} and {dk} are
generated using Algorithm 1, where the step size αk is computed via the SWP line search
such that the sufficient descent condition holds. If βk ≥ 0, there exists a constant γ > 0
such that ∥gk∥ > γ for all k ≥ 1. Then, dk ̸= 0 and

∞∑
k=0

∥uk+1 − uk∥2 < ∞, (20)

where uk = dk
∥dk∥ .

Proof. First, if dk = 0, then from the sufficient descent condition, we obtain gk = 0.
Thus, we suppose that dk ̸= 0 and

γ̄ ≥ ∥gk∥ ≥ γ > 0, ∀k ≥ 1. (21)

We now rewrite equation (10) as follows:

dFTCGLS
k = −gk +

(
−

gTk yk−1

dTk−1gk−1
− tk

gTk sk−1

dTk−1gk−1

)
dk−1 +

(
gTk dk−1

dTk−1gk−1

)
(yk−1 − sk−1).

We define
uk = wk + δkuk−1,

where

wk =
−gk + θk(yk−1 − sk−1)

∥dk∥
, δk = χ+

k

∥dk−1∥
∥dk∥

.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1440

Since uk is a unit vector, then

∥wk∥ = ∥uk − δkuk−1∥ = ∥δkuk − uk−1∥ .

Using the triangular inequality and δk ≥ 0,

∥uk − uk−1∥ ≤ (1 + δk) ∥uk − uk−1∥ = ∥uk − δkuk−1 − (uk−1 − δkuk)∥ , (22)

≤ ∥uk − δkuk−1∥+ ∥uk−1 − δkuk∥ = 2 ∥wk∥ .

We now define

ν = −gk − θk(yk−1 + sk−1).

Using the triangular inequality, we obtain

∥ν∥ ≤ ∥gk∥+ |θk| ∥yk−1 + sk−1∥ .

Moreover, using the equations of SWP line search, we can conclude that

|θk| =
∣∣gTk dk−1

∣∣∣∣dTk−1gk−1

∣∣ ≤ σ.

Now, using the triangular inequality and Assumption 1, we obtain

∥yk−1 + sk−1∥ ≤ ∥yk−1∥+ ∥sk−1∥ ≤ 2B + 2ρ.

Thus, the inequality in (4.3) can be written as follows:

∥ν∥ ≤ B + σ( 2B + 2ρ).

Let
H = B + σ( 2B + 2ρ),

then
∥ν∥ ≤ H.

From equation (22), we have
∥uk − uk−1∥ ≤ 2w.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1441

Thus, the following result is obtained

∞∑
k=0

∥uk+1 − uk∥2 ≤ 4
∞∑
k=0

∥w∥2 ≤ 4H2
∞∑
k=0

1

∥dk∥2
< ∞.

The proof is now complete. ■

The following property, which is referred to as Property*, was presented by Gilbert and
Nocedal in [15].

Property*
Consider a method of the form (2) and (3). Assume that (21) is satisfied for all k ≥ 1.
Then, the CG method has Property* if there exist constants b > 1 and λ > 0 such that
for all k ≥ 1, |χk| ≤ b and if ∥sk∥ ≤ λ, we obtain|ηk| ≤ 1

2b .

Lemma 4.3 Consider the CG method of the form (1) and (2) with χ+
k , where the step

size satisfies SWP line search (4) and (5). If equation (21) holds true, then χ+
k possesses

Property*. Namely, suppose (21) holds, then there exist b > 1and λ > 0 for all k ≥ 1
whereby

∣∣χ+
k

∣∣ ≤ b and if ∥sk∥ ≤ λ, we obtain|χk| ≤
1
2b .

Proof. As a result set b = 2(L+t)γ̄B
γ2 ≥ 1, andλ = γ2

2b(L+t)γ̄ .

Using SWP (4) and (5) with equation (21), we obtain∣∣χ+
k

∣∣ ≤ ∣∣∣∣∣ gTk yk−1

dTk−1gk−1

∣∣∣∣∣+t

∣∣∣∣∣ gTk sk−1

dTk−1gk−1

∣∣∣∣∣ ≤ L ∥gk∥ ∥sk−1∥+ t ∥gk∥ ∥sk−1∥
γ2

≤ 2(L+ t)γ̄B

γ2
= b > 1,

and if ∥sk∥ ≤ λ,

∣∣χ+
k

∣∣ ≤ ∣∣∣∣∣ gTk yk−1

dTk−1gk−1

∣∣∣∣∣+ t

∣∣∣∣∣ gTk sk−1

dTk−1gk−1

∣∣∣∣∣ ≤ L ∥gk∥ ∥sk−1∥+ t ∥gk∥ ∥sk−1∥
γ2

≤ (L+ t)γ̄λ

γ2
,

which implies ∣∣χ+
k

∣∣ ≤ 1

2b
(23)

The proof is complete. ■
The following Lemma and theorem are similar to that presented by [15]. Here, we present
Lemma 4.4 without its proof, which can be referred to in [15].



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1442

Lemma 4.4. Assume that Assumption 1 holds. Also, assume that the sequences
{gk} and {dk} are generated by Algorithm 1, in which αk is computed by the WWP
line search where the sufficient descent condition holds, assuming that the method has
Property*. Suppose also ∥gk∥ ≥ γ for someλ > 0. Then, there exists λ > 0 such that
for any∆ ∈ N and any index k0, there is an index k > k0 exists that satisfies∣∣∣κλk,∆∣∣∣ > λ

2
,

where κλk,∆ = {i ∈N : k ≤ i ≤ k +∆− 1, ∥si∥ > λ, N denotes the set of positive integers

and
∣∣∣κλk,∆∣∣∣ denotes the number of elements inκλk,∆.

Theorem 4.4 Suppose that Assumption 1 holds. Assume also that the sequences {gk}
and {dk} are generated by Algorithm 1 in which αk is computed by the WWP line search
and the sufficient descent condition holds. Also, suppose the Property* holds. Then, we
have lim

k→∞
inf ∥gk∥ = 0.

Proof. Based on Lemma 4.2 and Lemma 4.4, the proof is done by contradiction. We
defineui :=

di
∥di∥ . For any two indexes,l, k with l ≥ k, we have

xl − xk−1 =
l∑

i=k

∥si−1∥ui−1 ==
l∑

i=k

∥si−1∥uk−1 +
l∑

i=k

∥si−1∥ (ui−1 − uk−1),

where si−1 = xi − xi−1.
Taking the norms, we obtain

l∑
i=k

∥si−1∥ ≤ ∥xl∥+ ∥xk−1∥+
l∑

i=k

∥si−1∥ ∥ui−1 − uk−1∥ .

Using Assumption 1, we have that the sequence{xk} is bounded, and there exists a
positive constant ηsuch that ∥xk∥ ≤ η, for all k ≥ 1. Thus,

∥xl∥+ ∥xk−1∥ ≤ 2η,

which implies that

l∑
i=k

∥si−1∥ ≤ 2η +

l∑
i=k

∥si−1∥ ∥ui−1 − uk−1∥ . (24)

Assume λ > 0 given in Lemma 4.4, following the notation of this Lemma, we define

∆ :=

⌈
8η

λ

⌉
.

By Lemma 4.2, we can find an index k0such that



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1443

∞∑
k≥k0

∥ui − ui−1∥2 <
1

4∆
. (25)

With this ∆ and k0, Lemma 4.4 gives an index k ≥ k0 such that∣∣∣κλk,∆∣∣∣ > ∆

2
. (26)

Next, by the Cauchy-Schwarz inequality and (25), we have, for any index for i ∈ [k, k+
∆− 1] such that

∥ui−1 − uk−1∥ ≤
i−1∑
j=k

∥uj − uj−1∥ ,

≤ (i− k)1/2(
i−1∑
j=k

∥uj − uj−1∥2)1/2,

≤ ∆1/2(
1

4∆
)1/2 =

1

2
.

By this relation, (24) and (26), with l = k +∆− 1, we have

2η ≥ 1

2

k+∆−1∑
i=k

∥si−1∥ >
λ

2

∣∣∣κλk,∆∣∣∣ > λ∆

4
.

Thus, ∆ < 8η/λ, which contradicts the definition of ∆. The proof is then complete. ■

5. Numerical results

To test the efficiency of the new search direction (12), we selected some test functions in
Appendix 1 from CUTEr [21]. A comparison made with strong CG coefficients is such
as CG-Descent 5.3 [17], DL+[10], and the search direction dFTCGHS

k [1]. Since βDL
k is

not non-negative in general, we use βDL+
k similar to [24] as follows

βDL+
k = max{βHS

k , 0} − t
gTk sk−1

dTk−1yk−1
.βDL+

k = max{βHS
k , 0} − t

gTk sk−1

dTk−1yk−1
. (27)

Moreover, the authors in [10] restate equation (27) using the steepest descent if it does
not satisfy the descent condition. The comparison was made based on the CPU time,
number of iterations, number of function evaluations, and number of gradient evalu-
ations. We use SWP line search for dFTCGHS

k and DL+ method with δ = 0.01 and
σ = 0.1 similar to that used by the authors. For CG-Descent, we also used the Approxi-
mate Wolfe-Powell line search similar to that used by [17]. Moreover, we used SWP line
search for dFTCGLS

k similar to that used by [8] as follows:

f(xk + αkdk) ≤ f(xk) + δαkg
T
k dk,

σ1g
T
k dk ≤ g(xk + αkdk)

Tdk ≤ σ2g
T
k dk,



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1444

0 < δ < σ1 ≤ σ2 < 1,

where δ = 0.0001, σ1 = 0.1, and σ2 = 0.4.
The results of FTCGHS, FTCGLS, and DL+CG methods are obtained by running the
modified code of CG-Descent. The code can be obtained from the Hager webpage:

https://people.clas.ufl.edu/hager/software/

The norm of the gradient was employed as the stopping criterion, specifically ∥gk∥ ≤ 10−6

for all methods. The host computer is AMD A4-7210 APU Radeon R3 Graphics, where
the installed memory is 4 GB with operating system Ubuntu 20.04.2.0 LTS. The results
are shown in Figures 1, 2, and 3, in which a performance measure introduced by Dolan
and More [12] was employed. This performance measure was introduced to compare a
set of solvers S on a set of problems F. Assuming ns solvers and nf problems in S and
F, respectively, the measuretf,sis defined as the number of iterations or the CPU time
required to solve the problem f by the solver s. To create a baseline for comparison,
the performance of the solver son a problem f is scaled by the best performance of any
solver in S on the problem using the ratio

rf,s =
tf,s

min{tf,s : s ∈ S}
.

Suppose that a parameter rM ≥ rf,s for allf, s is chosen. Then, rf,s = rM if and only
if the solver s does not solve a problem f . Because we would like to obtain an overall
assessment of the performance of a solver, we defined the measure as

Ps(t) =
1

nf
size{f ∈ F : log rf,s ≤ t}.

Thus, Ps(t) is the probability for a solver s ∈ S that the performance ratio rf,s is
within a factor t ∈ R of the best possible ratio. Suppose we define the function psas
the cumulative distribution function for the performance ratio, then the performance
measure fs :R→ [0, 1] for a solver is non-decreasing and piecewise continuous from the
right. Thus, the value fs(1) is the probability that the solver has the best performance
of all the solvers. In general, a solver with high values of f(t), which would appear in
the upper right corner of the figure, is preferable.

5.1. Result analysis

In Appendix 1, the following big notations denote:
A: number of iterations.
B: number of function evaluations.
C: number of gradient evaluations.
D: CPU time.

Figure 1 shows that CG-Descent outperforms DL+ in terms of the number of iterations.
On the other hand, FTCGLS and FTCGHS outperform CG-Descent and DL+. Thus,



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1445

we can conclude that using the four-term CG method is better than using three-term CG
methods. In addition, we can note that FTCGLS slightly outperforms FTCGHS in the
number of iteration. From Figure 2, we can note that FTCGLS strongly outperforms all
methods in terms of the number of function evaluations. Thus, we conclude that using
extended SWP line search is better than using SWP line search. Figure 3 shows that
FTCGLS outperform FTCGHS, DL+, and CG-Descent in terms of CPU time. From
all figures, we can note that using four-term is better than using three-terms. Moreover,
using an extended SWP line search is better than using the original SWP line search
since the latter reduces the number of function evaluations. Finally, we can conclude
that FTCGLS is better than FTCGHS, CG-Descent, and DL+ in terms of efficiency.

Figure 1: Performance measure based on the number of iterations.



S. Ismail et al. / Eur. J. Pure Appl. Math, 14 (4) (2021), 1429-1456 1446

Figure 2: Performance measure based on the function evaluation.

Figure 3: Performance measure based on the CPU time.

Figure 4 presents the Diagonal 4 function in 3D. This function has a long narrow valley
with steep walls on both sides. Note that with dimension 2, the minimum is x∗ = (0, 0),
while the function value is f(x∗) = 0.



REFERENCES 1447

Figure 4: Diagonal 4 Function in 3D.

6. Conclusion

In this paper, we propose a new four-term CG method based on the LS CG method.
The new search direction satisfies the following properties:

(i) Equation (10) satisfies the descent property.

(ii) Equation (10) satisfies the convergence properties, i.e., the stationary point can be
obtained by the CG method with equation (10) for any unconstrained optimization
function.

(iii) Equation (10) contains a four-term CG method.

(iv) Equation (10) outperforms CG-Descent, DL+, and FTCGHS in the number of
iterations, function evaluations, and CPU time.

The future work will focus on the application of the CG methods in many fields such as
machine learning, deep learning, and regression problems.

Acknowledgements

The authors would like to thank the reviewers and editor for the comments that
highly improved this paper. Also, we would like to thank the University Sains Islam
Malaysia (USIM) for the support of this paper via grant P1-18-12220-UNI-RCR-FST.
Finally, the authors would like to thank Prof. Dr. William Hager for publishing the
code of CG-Descent.

References

[1] I. Masmali A. Alhawarat, G. Alhamzi and Z. Salleh. A Decent Four Term Conju-
gate Gradient Method with Global Convergence Properties for Large Scale Uncon-
strained Optimization Problems. Mathematical Problems in Engineering, 2021.

[2] M. Mamat A. Alhawarat, Z. Salleh and M. Rivaie. An efficient modified Po-
lak–Ribière–Polyak conjugate gradient method with global convergence properties.
Optimization Methods and Software, 32(6):1299–1312, 2017.



REFERENCES 1448

[3] Z. Salleh A. Alhawarat and I. A. Masmali. A Convex Combination between Two
Different Search Directions of Conjugate Gradient Method and Application in Image
Restoration. Mathematical Problems in Engineering, 2021.

[4] Z. Salleh A. Alhawarat and I. A. Masmali. A Convex Combination between Two
Different Search Directions of Conjugate Gradient Method and Application in Image
Restoration. Mathematical Problems in Engineering, 2021.

[5] M. Al-Baali. Descent property and global convergence of the Fletcher-Reeves
method with inexact line search. IMA J. Numer. Anal., 5:121–124, 1985.

[6] A. Alhawarat and Z. Salleh. Modification of nonlinear conjugate gradient method
with weak Wolfe-Powell line search. In Abstract and Applied Analysis, Hindawi,
2017.

[7] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press,
United Kingdom, 2004.

[8] E. K. P. Chong and S. H. Zak. An introduction to optimization. Wiley and Sons,
New York, 2nd ed, 2005.

[9] Y. Dai, D. Sun J. Han G. Liu, H. Yin, and Y. X. Yuan. Convergence properties of
nonlinear conjugate gradient methods. SIAM Journal on Optimization, 10(2):345–
358, 2000.

[10] Y. H. Dai and L. Z. Liao. New conjugacy conditions and related nonlinear conjugate
gradient methods. Appl. Math. Optim., 43(1):87–101, 2001.

[11] Y. H. Dai and Y. Yuan. A nonlinear conjugate gradient method with a strong
convergence property. SIAM Journal on Optimization, 10(1):177–182, 1999.

[12] E.D. Dolan and J.J. Moré. Benchmarking optimization software with performance
profiles. Math. Program., 91:201–213, 2002.

[13] R. Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.

[14] R. Fletcher and C. M. Reeves. Function minimization by conjugate gradients. The
Computer Journal, 7(2):149–154, 1964.

[15] J. C. Gilbert and J. Nocedal. Global convergence properties of conjugate gradient
methods for optimization. Siam. J. Optim., 2:21–42, 1992.

[16] I. C. Bivens H. Anton and S. Davis. Calculus: Early Transcendentals. John Wiley,
2010.

[17] W. W Hager and H. Zhang. A new conjugate gradient method with guaranteed
descent and an efficient line search. SIAM Journal on optimization, 16(1):170–192,
2005.



REFERENCES 1449

[18] W. W Hager and H. Zhang. The limited memory conjugate gradient method. SIAM
Journal on Optimization, 23(4):2150–2168, 2013.

[19] M. R. Hestenes and E. Stiefel. Methods of conjugate gradients for solving linear
systems. Journal of research of the National Bureau of Standards, 49(6):409–436,
1952.

[20] Z. Salleh I. A. Masmali and A. Alhawarat. A decent three-term conjugate gradient
method with global convergence properties for large scale unconstrained optimiza-
tion problems. AIMS Mathematics, 6(10):10742–10764, 2021.

[21] N. Gould I. Bongartz, A. R. Conn and P. L. Toint. CUTE: Constrained and
unconstrained testing environment. ACM Transactions on Mathematical Software
(TOMS), 21(1):123–160, 1995.

[22] L.Q. Zhang J. K. Liu, J.L. Xu. Partially symmetrical derivative-free Liu-Storey
projection method for convex constrained equations. Int. J. Comput. Math,
96:1787–1798, 2019.

[23] Y. Liu and C. Storey. Efficient generalized conjugate gradient algorithms, part 1:
theory. Journal of optimization theory and applications, 69(1):129–137, 1991.

[24] Y. Narushima M. Al-Baali and H. Yabe. A family of three-term conjugate gra-
dient methods with sufficient descent property for unconstrained optimization. .
Computational Optimization and Applications, 60(1):89–110, 2015.

[25] E. Polak and G. Ribiere. Note sur la convergence de méthodes de directions
conjuguées. ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation
Mathématique et Analyse Numérique, 3(R1):35–43, 1969.

[26] M. J. Powell. Non-convex minimization calculations and the conjugate gradient
method. In Numerical analysis, Springer, Berlin, Heidelberg:122–141, 1984.

[27] H. Tu S. Yao, L. Ning and J. Xu. A one-parameter class of three-term conjugate
gradient methods with an adaptive parameter choice. Optimization Methods and
Software, 35(6):1051–1064, 2020.

[28] J. A. Snyman. Practical mathematical optimization.. Springer Science and Business
Media, New York, 2005.

[29] W. Sun and Y. X. Yuan. Optimization theory and methods: nonlinear program-
ming. Springer Science & Business Media, 1, 2006.

[30] P. Wolfe. Convergence conditions for ascent methods. SIAM review, 11(2):226–235,
1969.

[31] P. Wolfe. Convergence conditions for ascent methods. II: Some corrections. SIAM
review, 13(2):185–188, 1971.



REFERENCES 1450

[32] I. Masmali Z. Salleh, G. Alhamzi and A. Alhawarat. A Modified Liu and Storey
Conjugate Gradient Method for Large Scale Unconstrained Optimization Problems.
Algorithms, 14(8):227, 2021.

[33] G. Zoutendijk. Nonlinear Programming, Computational Methods. Integer Nonlin-
ear Program, 143:37–86, 1970.



APPENDIX 1451

Appendix 1

FTCGLS FTCGHS CG-
DESCENT

DL+

Function A B C D A B C D A B C D A B C D

AKIVA 9 23 17 0.02 8 20 15 0.02 10 21 11 0.02 8 20 15 0.02

ALLINITU 9 23 16 0.02 9 25 18 0.02 12 29 18 0.02 9 25 18 0.03

ARGLINB 4 71 70 0.06 2 101 101 0.06 5 13 13 0.02 5 73 72 0.09

ARGLINC 3 33 32 0.05 2 98 98 0.08 11 106 110 0.02 5 79 78 0.06

BARD 11 29 21 0.02 12 32 22 0.02 16 33 17 0.02 12 32 22 0.02

BDEXP 2 8 8 0.02 2 7 7 0.02 5 11 6 0.02 2 7 7 0.02

BDQRTIC 113 236 203 0.52 154 310 302 0.65 136 273 237 0.52 168 363 359 0.63

BEALE 13 34 24 0.02 11 33 26 0.02 15 31 16 0.02 11 33 26 0.02

BIGGS3 80 190 124 0.02 79 207 144 0.02 110 231 125 0.02 79 207 144 0.02

BIGGS5 80 190 124 0.02 79 207 144 0.02 110 231 125 0.02 79 207 144 0.02

BIGGS6 23 57 39 0.02 24 64 44 0.02 27 57 31 0.02 24 64 44 0.02

BOX2 9 21 13 0.02 10 23 14 0.02 11 24 13 0.02 10 23 14 0.02

BOX3 9 21 13 0.02 10 23 14 0.02 11 24 13 0.02 10 23 14 0.02

BRKMCC 5 11 6 0.02 5 11 6 0.02 5 11 6 0.02 5 11 6 0.02

BROWNAL 6 17 13 0.02 8 19 12 0.02 9 25 18 0.02 10 29 21 0.02

BROWNBS 11 26 17 0.02 10 24 18 0.02 13 26 15 0.02 10 24 18 0.02

BROWNDEN 16 36 26 0.02 16 38 31 0.02 16 31 19 0.02 16 38 31 0.02

BROYDN7D 59 108 81 0.37 54 100 76 0.26 1411 2810 1429 5.22 75 138 112 0.36

BRYBND 35 98 72 0.22 32 86 62 0.15 85 174 90 0.28 149 317 174 0.55

CAMEL6 7 27 22 0.02 6 22 18 0.02 13 34 22 0.02 6 22 18 0.02

CHNROSNB 267 523 310 0.02 299 590 343 0.02 287 564 299 0.02 1009 1998 1180 0.01

CLIFF 6 45 37 0.02 10 46 39 0.02 18 70 54 0.02 10 46 39 0.01

CUBE 15 46 38 0.02 17 48 34 0.02 32 77 47 0.02 17 48 34 0.02

DENSCHNA 6 16 12 0.02 6 16 12 0.02 9 19 10 0.02 6 16 12 0.02

DENSCHNB 6 18 15 0.02 6 18 15 0.02 7 15 8 0.02 6 18 15 0.02

DENSCHNC 15 44 36 0.02 11 36 31 0.02 12 26 14 0.02 11 36 31 0.02

DENSCHND 15 50 44 0.02 14 46 40 0.02 47 98 51 0.02 14 46 40 0.02

DENSCHNE 13 42 35 0.02 12 43 38 0.02 18 49 32 0.02 12 43 38 0.02



APPENDIX 1452

FTCGLS FTCGHS CG-
DESCENT

DL+

DENSCHNF 10 30 24 0.02 9 31 26 0.02 8 17 9 0.02 9 31 26 0.02

DIXMAANA 6 16 13 0.02 8 19 13 0.02 7 15 8 0.02 6 15 11 0.02

DIXMAANB 7 16 10 0.02 19 111 110 0.05 6 13 7 0.02 6 15 11 0.02

DIXMAANC 6 14 9 0.02 20 135 134 0.09 6 13 7 0.02 6 14 9 0.02

DIXMAAND 8 20 14 0.02 26 153 148 0.08 7 15 8 0.02 7 17 12 0.02

DIXMAANE 251 279 482 0.28 254 286 481 0.28 222 239 429 0.23 394 428 764 0.5

DIXMAANF 171 347 179 0.16 174 352 179 0.16 161 323 162 0.17 247 499 255 0.27

DIXMAANG 165 334 171 0.16 165 336 174 0.22 157 315 158 0.12 348 701 356 0.38

DIXMAANH 152 311 163 0.13 162 334 177 0.14 173 347 174 0.2 332 671 343 0.45

DIXMAANI 2856 2917 5659 3.19 2811 2893 5548 3.63 3856 3926 7644 4.09 3522 3623 6953 4.66

DIXON3DQ 10000 10007 19995 20.3 10000 10007 19995 20.3 10000 10007 19995 19.48 15258 15265 30511 37.63

DJTL 93 1469 1446 0.02 75 1163 1148 0.02 82 917 880 0.02 75 1163 1148 0.02

DQDRTIC 5 11 6 0.02 5 11 6 0.02 5 11 6 0.02 15 32 18 0.02

ECKERLE4LS.SIF 3 7 4 0.02 2 6 4 0.02 3 7 4 0.02 2 6 4 0.02

EDENSCH 26 59 50 0.05 30 81 74 0.05 26 52 38 0.02 27 66 54 0.03

EGGCRATE 6 15 10 0.02 6 15 10 0.02 6 15 10 0.02 6 15 10 0.02

EIGENALS 7318 12590 9382 155.7 10197 18439 12170 172.55 10083 18020 12244 172.67 9534 18450 18540 185.64

ENGVAL1 22 48 39 0.06 24 53 46 0.08 27 50 36 0.06 21 48 37 0.06

ENGVAL2 25 69 52 0.02 26 73 55 0.02 26 61 37 0.02 26 73 55 0.02

ENSOLS 21 45 27 0.02 22 47 27 0.02 23 45 26 0.02 22 47 27 0.02

EXPFIT 10 29 21 0.02 3925 7575 5345 0.11 13 29 16 0.02 9 29 22 0.02

exp2 6 14 8 0.02 7 16 9 0.02 8 17 9 0.02 7 16 9 0.02

FBRAINLS 10 29 23 0.03 9 27 21 0.03 10 23 14 0.02 9 27 21 0.02

FBRAIN2LS 95 285 220 0.58 79 259 204 0.47 118 339 248 0.66 79 259 204 0.47

FMINSRF2 304 628 332 1.09 718 1133 1771 3.47 346 693 347 0.97 733 1545 826 2.34

FMINSURF 452 925 477 1.59 654 905 1248 2.84 473 947 474 1.42 1245 2567 1342 4.08

GENHUMPS 9395 19074 9741 0.37 8 107 106 0.37 6475 12964 6493 19.89 4938 14763 10180 25.27

GROWTHLS 107 382 315 0.02 109 431 369 0.02 156 456 319 0.02 109 431 369 0.02

GULF 27 90 70 0.02 33 95 72 0.02 37 84 48 0.02 33 95 72 0.02



APPENDIX 1453

FTCGLS FTCGHS CG-
DESCENT

DL+

HAHN1LS 4 54 51 0.02 5 56 53 0.02 37 121 86 0.02 5 56 53 0.02

HAIRY 16 63 51 0.02 17 82 68 0.02 36 99 65 0.02 17 82 68 0.02

HATFLDD 17 47 38 0.02 17 49 37 0.02 20 43 24 0.02 17 49 37 0.02

HATFLDE 12 33 25 0.02 13 37 30 0.02 30 72 45 0.02 13 37 30 0.02

HATFLDFL 39 123 100 0.02 21 68 54 0.02 39 92 55 0.02 21 68 54 0.02

HATFLDFLS 53 157 124 0.02 48 156 125 0.02 64 155 97 0.02 48 156 125 0.02

HEART6LS 376 1083 814 0.02 375 1137 876 0.02 684 1576 941 0.02 375 1137 876 0.02

HEART8LS 240 609 414 0.02 253 657 440 0.02 249 524 278 0.02 253 657 440 0.02

HELIX 23 59 42 0.02 23 60 42 0.02 23 49 27 0.02 23 60 42 0.02

HIELOW 13 30 22 0.05 13 30 21 0.03 14 30 16 0.02 13 30 21 0.05

HILBERTA 2 5 3 0.02 2 5 3 0.02 2 5 3 0.02 2 5 3 0.02

HILBERTB 4 9 5 0.02 4 9 5 0.02 4 9 5 0.02 4 9 5 0.02

HIMMELBB 4 18 17 0.02 4 18 18 0.02 10 28 21 0.02 4 18 18 0.02

HIMMELBF 23 56 40 0.02 23 59 46 0.02 26 60 36 0.02 23 59 46 0.02

HIMMELBG 7 22 17 0.02 7 22 17 0.02 8 20 13 0.02 7 22 17 0.02

HIMMELBH 5 13 9 0.02 5 13 9 0.02 7 16 9 0.02 5 13 9 0.02

HUMPS 36 226 205 0.02 45 223 202 0.02 52 186 146 0.02 45 223 202 0.02

HYDCAR6LS.SIF 70 143 74 0.02 120 242 123 0.02 14401 29028 14875 0.45 1001 2027 1174 0.03

INTEQNELS.SIF 6 13 7 0.02 7 15 8 0.02 6 13 7 0.02 6 13 7 0.02

JENSMP 15 54 45 0.02 12 47 41 0.02 15 33 22 0.02 12 47 41 0.02

JUDGE 9 24 17 0.02 9 24 18 0.02 10 23 13 0.02 9 24 18 0.02

LANCZOS1LS 73 184 129 0.02 61 177 135 0.02 148 325 181 0.02 61 177 135 0.02

LANCZOS2LS 70 175 119 0.02 60 169 125 0.02 169 379 215 0.02 60 169 125 0.02

LANCZOS3LS 70 177 125 0.02 61 164 118 0.02 179 392 219 0.02 61 164 118 0.02

LOGHAIRY 14 91 79 0.02 26 196 179 0.02 27 81 58 0.02 26 196 179 0.02

LSC1LS 34 106 85 0.02 31 108 89 0.02 36 101 71 0.02 31 108 89 0.02

LSC2LS 55 173 141 0.02 37 106 86 0.02 54 119 67 0.02 37 106 86 0.02

LUKSAN13LS 90 178 142 0.02 90 182 168 0.02 84 158 121 0.02 142 279 243 0.02

LUKSAN14LS 156 324 213 0.02 188 400 254 0.02 98 122 156 0.02 157 313 201 0.02



APPENDIX 1454

FTCGLS FTCGHS CG-
DESCENT

DL+

LUKSAN15LS 27 61 45 0.02 27 61 47 0.02 28 59 44 0.02 27 60 45 0.02

LUKSAN16LS 28 56 38 0.02 28 55 38 0.02 31 57 38 0.02 35 72 53 0.02

MANCINO 10 21 11 0.02 12 29 18 0.09 11 23 12 0.06 11 23 12 0.06

MEXHAT 17 64 60 0.02 14 59 55 0.02 20 56 39 0.02 14 59 55 0.02

MEYER3 145 611 534 0.02 19 76 63 0.02 19 67 52 0.02 19 76 63 0.02

MGH09LS 49 143 110 0.02 25 82 72 0.02 57 137 86 0.02 25 82 72 0.02

MGH10LS 1084 4530 5502 0.02 1082 4052 4968 0.03 1134 4464 535 0.03 108 4052 4968 0.02

MGH10SLS 73 470 437 0.02 19 112 102 0.03 146 505 401 0.03 19 112 102 0.02

MGH17LS 58 199 150 0.02 84 323 265 0.03 228 564 363 0.02 84 323 365 0.02

MISRA1BLS.SIF 13 53 62 0.02 26 113 101 0.03 35 139 117 0.02 26 113 101 0.02

MISRA1CLS.SIF 24 82 94 0.02 26 145 121 0.03 26 110 91 0.02 26 145 121 0.02

MISRA1DLS.SIF 21 74 74 0.02 22 90 84 0.02 24 74 75 0.02 22 90 84 0.02

MOREBV 161 168 317 0.39 161 168 317 0.31 161 168 317 0.42 117 124 229 0.23

MSQRTBLS 2201 4410 2211 5.5 2296 4392 2514 8.03 2280 4525 2326 6.91 5786 11558 5818 17.72

NELSONLS 1117 3861 5389 0.17 1101 5415 7690 0.2 1118 5692 7331 0.17 1101 5415 7690 0.23

NONDIA 7 25 19 0.02 7 17 11 0.01 7 25 20 0.03 7 25 19 0.03

OSBORNEA 67 174 122 0.02 82 230 174 0.02 94 213 124 0.02 82 230 174 0.02

OSBORNEB 58 140 91 0.01 57 134 84 0.01 62 127 65 0.02 57 134 84 0.02

OSCIPATH 292090 7E+05 5E+05 1.3 2950298E+055344252.42 3109906709533673251.91 3E+057817295344252.42

PALMER1C 11 26 26 0.02 12 27 28 0.02 11 26 26 0.02 12 27 28 0.02

PALMER1D 10 26 26 0.02 10 24 23 0.02 11 25 25 0.02 10 24 23 0.02

PALMER2C 11 20 20 0.02 11 21 22 0.02 11 21 21 0.02 11 21 22 0.02

PALMER3C 11 20 20 0.02 11 21 21 0.02 11 20 20 0.02 11 21 21 0.02

PALMER4C 11 20 20 0.02 11 21 21 0.02 11 20 20 0.02 11 21 21 0.02

PALMER5C 6 13 7 0.02 6 13 7 0.02 6 13 7 0.02 6 13 7 0.02

PALMER6C 11 25 26 0.02 11 24 24 0.02 11 24 24 0.02 11 24 24 0.02

PALMER7C 11 21 22 0.02 11 20 20 0.02 11 20 20 0.02 11 20 20 0.02

PALMER8C 11 21 22 0.02 11 19 0.02 11 18 17 0.02 11 19 19 0.02

PENALTY1 20 63 53 0.02 2 28 28 0.02 28 69 44 0.02 14 51 43 0.02

PENALTY2 185 220 351 0.02 191 225 369 0.02 191 221 354 0.03 337 480 758 0.06

PENALTY3 79 247 201 0.83 51 148 110 0.84 99 285 219 1.74 102 346 290 2.19



APPENDIX 1455

FTCGLS FTCGHS CG-
DESCENT

DL+

POWELLBSLS 38 148 123 0.02 50 211 234 0.02 61 247 246 0.02 50 211 234 0.02

POWELLSG 26 63 44 0.02 30 83 63 0.05 26 53 27 0.03 36 92 65 0.05

0.4 357 731 382 0.7 355 724 376 0.59 372 754 384 0.58 356 733 391 0.58

POWERSUM 5 11 6 0.02 4 10 6 0.59 5 11 6 0.02 4 10 6 0.02

PRICE3 11 26 18 0.02 10 25 17 0.02 11 23 12 0.02 10 25 17 0.02

PRICE4 10 28 21 0.02 9 30 23 0.02 16 32 18 0.02 9 30 23 0.02

QING 68 135 87 0.03 67 134 85 0.03 68 135 84 0.02 85 179 96 0.02

QUARTC 15 32 18 0.02 37 167 155 0.08 17 37 21 0.02 15 32 18 0.02

RAT43LS.SIF 5 55 317 0.02 44 156 122 0.02 54 145 97 0.02 44 156 122 0.02

RECIPELS.SIF 14 36 26 0.02 16 49 38 0.02 27 58 32 0.02 16 49 38 0.02

ROSENBR 24 77 61 0.02 28 84 65 0.02 34 77 44 0.02 28 84 65 0.02

ROSENBRTU.SIF 39 166 141 0.02 37 175 153 0.02 49 156 113 0.02 37 175 153 0.02

S308 7 20 16 0.02 7 21 17 0.02 8 19 12 0.02 7 21 17 0.02

SCHMVETT 43 74 59 0.02 38 69 52 0.17 43 73 60 0.02 59 103 88 0.28

SENSORS 21 57 41 0.02 25 68 47 0.38 21 50 34 0.02 24 71 53 0.47

SINEVAL 41 155 129 0.02 46 181 153 0.02 64 144 88 0.02 46 181 153 0.02

SINQUAD 14 40 32 0.08 14 43 34 0.08 14 40 33 0.08 13 46 38 0.09

SISSER 6 22 21 0.02 5 19 19 0.08 6 18 14 0.02 5 19 19 0.02

SNAIL 15 48 36 0.02 61 251 211 0.02 100 230 132 0.02 61 251 211 0.02

SROSENBR 9 23 16 0.02 9 25 19 1.14 11 23 12 0.02 9 23 15 0.02

SSI 276 892 1063 0.02 307 1162 990 0.02 345 948 657 0.02 307 1162 990 0.02

STREG 58 184 146 0.02 60 218 180 0.02 96 224 139 0.02 60 218 180 0.02

STRATEC 158 353 232 5.58 170 419 283 6.33 462 1043 796 19 170 419 283 6.3

STRTCHDV.SIF 11 36 33 0.02 12 38 32 0.02 16 35 20 0.02 12 38 32 0.02

TESTQUAD 1573 1580 3141 1.2 1580 1587 3155 1.52 1577 1584 3149 1.28 20325 20361 40674 21.61

THURBERLS 84 213 146 0.02 105 259 216 0.02 102 232 175 0.02 105 259 216 0.02

TOINTGOR 122 216 154 0.02 118 216 154 0.02 135 233 174 0.02 192 348 270 0.02

TOINTGSS 4 9 5 0.02 4 10 7 0.02 4 9 5 0.02 4 9 5 0.02

TOINTPSP 162 343 259 0.02 151 319 250 0.02 143 279 182 0.02 145 313 250 0.02

TOINTQOR 29 36 53 0.02 29 36 53 0.02 29 36 53 0.02 49 56 93 0.02

TQUARTIC 11 39 32 0.02 13 45 37 0.03 14 40 27 0.02 11 41 34 0.03

TRIDIA 780 787 1555 0.75 780 787 1555 1.03 782 7889 155 0.89 469 4721 9408 6.38



APPENDIX 1456

FTCGLS FTCGHS CG-
DESCENT

DL+

TRIGON1.SIF 19 40 21 0.02 19 41 22 0.02 22 45 23 0.02 19 41 22 0.02

TRIGON2.SIF 25 58 37 0.02 22 57 43 0.02 26 52 28 0.02 22 57 43 0.02

VANDANMSLS.SIF8 27 20 0.02 8 25 18 0.02 5 12 7 0.02 5 11 6 0.02

VARDIM 7 20 17 0.02 1 4 4 0.02 10 21 11 0.02 9 20 15 0.02

VAREIGVL 24 51 29 0.02 27 60 37 0.02 23 47 21 0.02 28 71 51 0.02

VESUVIALS 1136 1496 2821 0.02 1262 1954 3155 0.02 1519 2317 4111 1.56 1262 1954 3155 1.22

VESUVIOULS 72 168 117 0.02 79 211 173 0.02 80 180 131 0.06 79 211 173 0.09

VIBRBEAM 233 552 415 0.02 98 255 174 0.02 138 323 199 0.02 98 255 174 0.02

WAYSEA1 12 49 41 0.02 11 55 50 0.02 18 39 22 0.02 11 55 50 0.02

WAYSEA2 9 28 23 0.02 9 28 23 0.02 31 68 39 0.02 9 28 23 0.02

WOODS 26 65 43 0.05 68 184 129 0.02 22 51 30 0.03 24 62 41 0.03

YATP1CLS 17 48 36 5.3 14 41 31 5.76 23 53 31 6.13 17 48 36 7.12

YFITU 66 200 159 0.02 68 208 167 0.02 84 197 118 0.02 68 208 167 0.03

ZANGWIL2 1 3 2 0.02 1 3 2 0.02 1 3 2 0.02 1 3 2 0.02