412 
 

IHJPAS. 37 (1) 2024 

Ibn Al-Haitham Journal for Pure and Applied Sciences 

Journal homepage: jih.uobaghdad.edu.iq 

PISSN: 1609-4042, EISSN: 2521-3407 

  

 

 

 

 

  

1Seror Faeq Mohammed    2Ghadeer Jasim Mohammed Mahdi*    

3Md Kamrul Hasan Khan    
1,2Department of Mathematics,College of Education for the Pure Science Ibn Al-Haitham , University of 

Baghdad, Iraq,. 

3Department of Mathematical Sciences, Fulbright College of Arts & Sciences, 1University of 

Arkansas, USA 

*Corresponding Author. gmahdi@ihcoedu.uobaghdad.edu.iq, 

 

 

 

 

Abstract   

Support Vector Machines (SVMs) are supervised learning models used to examine data sets in 

order to classify or predict dependent variables. SVM is typically used for classification by 

determining the best hyperplane between two classes. However, working with huge datasets can 

lead to a number of problems, including time-consuming and inefficient solutions. This research 

updates the SVM by employing a stochastic gradient descent method. The new approach, the 

extended stochastic gradient descent SVM (ESGD-SVM), was tested on two simulation datasets. 

The proposed method was compared with other classification approaches such as logistic 

regression, naive model, K Nearest Neighbors and Random Forest. The results show that the 

ESGD-SVM has a very high accuracy and is quite robust. ESGD-SVM is used to analyze the heart 

disease dataset downloaded from Harvard Dataverse. The entire analysis was performed using the 

program R version 4.3. 

 

Keywords SVM, classification, reduction of dimensions, variables selection, gradient descent, 

heart disease. 

 

 1. Introduction 

Suppose we have a data set where for each subject we have information about an 𝑝-

dimensional covariate vector , 𝑥𝑝×1 and a response y that has two possible categories. Our goal is 

to develop an algorithm that allows us to predict for each new observation the category of its 

response based on its 𝑝-information. The Support Vector Machine (SVM) is one of the many 

methods to do this, such as: Logistic Regression [1], Random Forest [2], k nearest neighbors [3], 

Naïve Bayes [4] and LDA [5]. SVM is a supervised learning technique, i.e., we create a classifier 

based on a training dataset and use this classifier for future observations. 

Received 7 May 2023, Received 8 June 2023, Accepted 12 June 2023, Published 20 January 2024 

 

 

Enhanced Support Vector Machine Methods Using Stochastic Gradient Descent and 

Its Application to Heart Disease Dataset 
 

doi.org/10.30526/37.1.3467 

https://jih.uobaghdad.edu.iq/index.php/j/index#1609-4042
https://jih.uobaghdad.edu.iq/index.php/j/index#2521-3407
mailto:gmahdi@ihcoedu.uobaghdad.edu.iq
https://orcid.org/0000-0002-4444-4362
mailto:serorfaeqmohammed@gmail.com
https://orcid.org/0000-0003-4870-4034
mailto:gmahdi@ihcoedu.uobaghdad.edu.iq,
https://orcid.org/
mailto:mkk006@uark.edu


IHJPAS. 37 (1) 2024 

413 
 

For group one, let 𝑦 = +1 and for group two, let 𝑦 = −1. The goal of SVM is to design a 

classifier 𝑓(𝑥) such that, the classifier rule: 𝑦 = +1 if 𝑓(𝑥)  >  0 and 𝑦 = −1 if 𝑓(𝑥) <  0, can 

be used to determine the response category given the covariate information. Unlike LDA, SVM 

does not use any distribution for 𝑥 given its category. Instead, it is a geometric procedure that 

finds the classifier according to some optimization criterion. 

In this work, we will discuss linear and nonlinear SVMsSuppose 𝑓(𝑥)=,𝛽-0.+,𝛽-𝑇.𝑋 for 

unknown parameters (,𝛽-0.,𝛽) is a function, a hyperplane in the space, that acts as a separator 

between the two response categories. . 

Linearity is a simplifying assumption, and it is not reasonable to always assume that ƒ is linear, 

because in general, non-linear SVMs are a popular tool for many real-world applications. In some 

cases, linear separation may work sufficiently so that we do not need to consider nonlinear 

assumptions. 

We organize the paper as follows: In Sections 2 and 3, we introduce the hard (linear) boundary 

and the soft (nonlinear) boundary. The kernel transformation and its properties are discussed in 

Section 4. The improved SVM with stochastic gradient descent is explained in Section 5. 

Simulation studies with three data sets and the application to heart disease are described in Sections 

6 and 7, respectively. Finally, the results and discussion are discussed in Section 8. 

 

2.  Hard Margin 

Assume that two categories are linearly separable, so there exists a hyperplane 𝑓(𝑥) = 𝛽0 +

𝛽𝑇 = 0 that separates the categories. Our task is to find out estimates of 𝛽0 and 𝛽. Suppose 

(𝑥(1), 𝑦1), (𝑥(2), 𝑦2), . . . (𝑥(𝑛), 𝑦𝑛) are 𝑛 data points from above setting. Consider any hyper plane 

𝛽0 + 𝛽𝑇 = 0 in the x-space. The perpendicular distance of the 𝑖𝑡ℎ covariate point 𝑥(𝑖) from this 

line is 𝑑𝑖 = 𝑦𝑖
𝛽0+𝛽𝑇𝑋(𝑖)

ǁ𝛽ǁ
 , and for all training data points 𝑦𝑖(𝛽0 + 𝛽𝑖𝑥) > 0 [6]. The minimum of 

these distance is called the margin, i.e., there is no data point within this distance on either side of 

this line with respect to this training dataset [7]. 

The aim is to find a line that has maximum margin among all candidate lines. That line is 

going to be our estimate of classifier 𝑓(𝑋). The intuition behind keeping the margin maximum is 

a supervised learning procedure. We want to ensure that, for new observations (test data), we 

accurately determine category of 𝑦. We want to maximize the margin because it gives us a room 

for allowing for variation between training and test datasets. 

 

Assume the optimization problem is 

𝑚𝑖𝑛𝛽0,𝛽
ǁ𝛽ǁ2

2
     subject to 𝑦𝑖(𝛽0 + 𝛽𝑇𝑥(𝑖)) ≥ 1 𝑓𝑜𝑟 𝑖 = 1,2, . . . , 𝑛   (1) 

If (�̂�0, �̂�) represent the solution to this optimization problem, some statements according to eq(1) 

can be hold. 

First, ƒ̂(𝑥) = �̂�0 + �̂�𝑇𝑋 = 0 represents the estimated linear classifier. For any new data point with 

covariate information 𝑥0, evaluate ƒ̂(𝑥0) = �̂�0 + �̂�𝑇𝑋0. If ƒ̂(𝑥0) > 0, decide category1 for that 



IHJPAS. 37 (1) 2024 

414 
 

observation, otherwise if ƒ̂(𝑥0) < 0, decide category 2. Second, the margin of the classifier is: 
1

ǁ�̂�ǁ
 

. Geometrically, there will not be any training data points between the lines 

�̂�0 + �̂�𝑇𝑋 = +1 and �̂�0 + �̂�𝑇𝑋 = −1. 

As shown in Fig.1, many lines can separate two groups ( 𝐻1, 𝐻2, 𝐻3, and 𝐻4), but there is only 

one optimal separating hyperplane (𝐻) with two boundary hyperplanes (𝐻𝑚1 𝑎𝑛𝑑 𝐻𝑚2) see Fig.2. 

Third, the training data points lie exactly on one of the above two lines (margins) are referred to 

as support vectors, so the estimates �̂�0, 𝛽 depend only on the data points that are support vectors. 

That means all data points inside the correct margins have no role in determining the form of the 

classifier. This property makes SVM very useful for massive data classification problems [7]. 

 

 

Figure 1. Some lines that separate two groups: 𝐻1, 𝐻2, 𝐻3, 𝑎𝑛𝑑 𝐻4. 



IHJPAS. 37 (1) 2024 

415 
 

 

Figure 2. separating hyperplane, 𝐻, with two boundary hyperplanes : 𝐻𝑚1 𝑎𝑛𝑑 𝐻𝑚2. 

 

3. Soft Margin 

Suppose the assumption of linear separability does not hold between the two categories. That 

implies, no matter what hyper plane 𝛽0 + 𝛽𝑇𝑋 = 0, there will always be points (𝑥, 𝑦) such that: 

𝛽0 + 𝛽𝑇𝑥 > 0 𝑏𝑢𝑡 𝑦 = −1 𝑎𝑛𝑑 𝛽0 + 𝛽𝑇𝑥 < 0 𝑏𝑢𝑡 𝑦 = +1. 

Consider a nonnegative variable 𝐾𝑖 for each observation I =1,2, …, n. 𝐾𝑖 denotes the amount of 

push an observation needs to go the correct side of the margin. We have shown in class that [8]: 

a. For points already obeying correct margin, 𝐾𝑖 = 0. 

b. For points violating the margin but staying on the correct side of the line, 0 < 𝐾𝑖 <
1

ǁ𝛽ǁ
. 

c. For points violating the margin and moving to the other side of the line    𝐾𝑖 >
1

ǁ𝛽ǁ
. 

By solving the optimization problem, the estimates of (𝛽0, 𝛽) can be found: 

𝑚𝑖𝑛
𝛽0,𝛽, 𝑠𝑖

ǁ𝛽ǁ2

2
+ 𝐶 ∑ 𝑠𝑖

𝑛

𝑖=1

  𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑦𝑖(𝛽0 + 𝛽𝑇𝑋(𝑖)) ≥ 1 − 𝑠𝑖 , 𝑠𝑖 ≥ 0 𝑓𝑜𝑟 𝑖

= 1,2, . . . , 𝑛 

   (2) 

In eq (2), C is a pre-fixed turning parameter that balances the relative importance of maximizing 

the margin and minimizing the total amount of push required for points violating the margin. 

If (�̂�0, 𝛽) represent the solution to this optimization problem, then the following statements hold: 

1. 𝑓(𝑥) = �̂�0 + �̂�𝑇𝑥 = 0 represents the estimated linear classifier. For any new data point with 

covariate information 𝑥0, evaluate 𝑓(𝑥0) = �̂�0 + �̂�𝑇𝑥0. 𝐼𝑓 𝑓(𝑥0) > 0, decide category 1 for that 

observation, otherwise if 𝑓(𝑥0) < 0, decide category 2. 



IHJPAS. 37 (1) 2024 

416 
 

2.The Margin of the classifier is: 
1

ǁ𝛽ǁ̂
. 

3. In machine learning literature {𝑠𝑖} are called slack variables. 

4. In this case, the support vectors are defined as those observations in the training data lies exactly 

on any one of the two correct margins, 𝑦𝑖(β̂0 + β̂𝑇𝑥(𝑖)) = +1, or violate the correct margins, 

𝑦𝑖(β̂0 + β̂𝑇𝑥(𝑖)) < +1. 

5. It turns out that the estimations β̂0, β̂ depend only on the data points which are support vectors. 

That means all data points inside the correct margins have no role in determining the form of the 

classifier. This property makes SVM very useful for massive data classification problems. 

For any two nonlinearly separable classes (e.g., because of the noise), the optimal hyper-plane 

condition including an extra term can be formalized as follows: 

𝒚𝒊(𝑥𝑖
𝑇w + b) ≥ 1 − 

𝑖
,        , 𝑖 = 1, . . . , 𝑛. 

The objective function should be minimized, i.e., 
𝑖

≥ 0 should be minimized as well as ǁ𝑤ǁ, as 

follows: 

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒   𝑤𝑇𝑤 + 𝑐 ∑ 
𝑖
𝑘𝑛

𝑖=1  subject to 𝑦𝑖(𝑥𝑖
𝑇𝑤 + 𝑏) ≥ 1 − 

𝑖
 and 

𝑖
≥ 0; 𝑖 =

1, . . . , 𝑛. 
   (3) 

In eq (3), C is a regularization parameter that controls the balance between making the margin 

as big as possible and making the training error as small as possible. Small 𝐶 tends to focus on the 

margin and ignore outliers in the training data, while large 𝐶 may cause the training data to be too 

well fit (see Figure 3). 

3.1 Second Norm Soft Margin 

The optimization problem is called  the second soft (nonlinear) margin problem when 𝑘 = 2, i.e., 

𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒    𝑤𝑇𝑤 + 𝑐 ∑ 
𝑖
2

𝑚

𝑖=1

. 

subject to  𝑦𝑖(𝑥𝑖
𝑇𝑤 + 𝑏) ≥ 1 − 

𝑖
,   (𝑖 = 1, . . . , 𝑚). 

Not that if 
𝑖

≥ 0 is declined, similarly if 
𝑖

< 0, we can set it to zero and the above function 

can be further reduced. 

The initial Lagrangian for the above 2-norm problem is 

𝐿𝑝(𝑤, 𝑏, , 𝛼) =
1

2
𝑤𝑇𝑤 +

𝑐

2
∑ 

𝑖
2

𝑛

𝑖=1

− ∑ 𝛼𝑖

𝑛

𝑖=1

[𝑦𝑖(𝑤𝑇𝑥 + 𝑏) − 1 + 
𝑖
]. 

Substituting 

𝜕𝐿

𝜕𝑤
= w − ∑ 𝑦𝑖𝛼𝑖𝑥𝑖

𝑛

𝑖=1

= 0;  
𝜕𝐿

𝜕
= 𝑐 − 𝛼 = 0;  

𝜕𝐿

𝜕𝑏
= ∑ 𝑦𝑖𝛼𝑖

𝑛

𝑖=1

= 0. 

Into the initial Lagrangian, we get the dual problem  



IHJPAS. 37 (1) 2024 

417 
 

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝐿𝑑(𝛼) = ∑ 𝛼𝑖

𝑚

𝑖=1

−
1

2
∑ ∑ 𝑦𝑖𝑦𝑗𝛼𝑖𝛼𝑗𝑥𝑗

𝑇𝑥𝑖 −
1

2𝑐
∑ 𝛼𝑖

2

𝑚

𝑖=1

𝑚

𝑗=1

𝑚

𝑖=1

 

= ∑ 𝛼𝑖

𝑚

𝑖=1

−
1

2
∑ ∑ 𝑦𝑖𝑦𝑗𝛼𝑖𝛼𝑗

𝑚

𝑗=1

𝑚

𝑖=1

(𝑥𝑗
𝑇𝑥𝑖 +

1

𝑐
𝛿𝑖𝑗). 

Subject to     𝛼𝑖 ≥ 0, ∑ 𝛼𝑖
𝑚
𝑖=1 𝑦𝑖 = 0. 

The above quadratic programming can be solved for 𝛼𝑖. As a result, all support vectors that 

correspond to 𝑥𝑖 > 0 satisfy the following condition [8]: 

𝑦𝑖(𝑋𝑖
𝑇𝑤 + 𝑏) = 1 − 𝜉𝑖 . 

Substituting   𝑤 = ∑ 𝑦𝑗𝑦𝑗𝑥𝑗𝑗𝜖𝑠  into this equation (where S is the set of support vector), we get:  

𝑦𝑖(∑ 𝑦𝑗𝛼𝑗(𝑋𝑖
𝑇𝑋𝑗) + 𝑏𝑗∈𝑆 ) = 1 − 𝜉𝑖, 𝑖. 𝑒. ,   𝑦𝑖 ∑ 𝑦𝑗𝛼𝑗𝑗∈𝑆 (𝑥𝑖

𝑇𝑥𝑗) = 1 − 𝜉𝑖 − 𝑦𝑖𝑏 . 

For the optimal weight w, we have : 

ǁ𝑤ǁ2  ∶  𝑤𝑇𝑤 = ∑ 𝛼𝑖𝑦𝑖𝑥𝑖
𝑇

𝑖∈𝑆

∑ 𝛼𝑗𝑦𝑗𝑥𝑗

𝑗∈𝑆

= ∑ 𝛼𝑖𝑦𝑖 ∑ 𝛼𝑖𝑦𝑖𝑥𝑖
𝑇𝑥𝑗

𝑗∈𝑆𝑖∈𝑆

. 

∑ 𝛼𝑖(1 − 𝜉𝑖 − 𝑦𝑖
𝑏) = ∑ 𝛼𝑖 − ∑ 𝛼𝑖𝜉𝑖 − 𝑏 ∑ 𝑦𝑖𝛼𝑖𝑖∈𝑆𝑖∈𝑆𝑖∈𝑆𝑖∈𝑆 . 

Since 𝜉𝑖 = 𝛼𝑖/𝐶, we have: 

∑ 𝛼𝑖𝑖∈𝑆 − ∑ 𝛼𝑖𝜉𝑖 = ∑ 𝛼𝑖 −
1

𝑐
∑ 𝛼𝑖

2
𝑖∈𝑆𝑖∈𝑆𝑖∈𝑆 . 

Therefor the optimal separation margin becomes: 

 
1

ǁwǁ
= (∑ 𝛼𝑖 −

1

𝑐
∑ 𝛼𝑖

2
𝑖∈𝑆𝑖∈𝑆 )−1/2. 

 

3.2 First Norm Soft Margin 

The optimization problem is called the first soft (nonlinear) margin problem when k =1, i.e., 

minimize 𝑤𝑇𝑤 + 𝑐 ∑ 
𝑖

𝑚
𝑖=1  

Subject to   𝑦𝑖(𝑥𝑖
𝑇w + b) ≥ 1 − 

𝑖
, 

𝑖
≥ 0; i=1,…, n. 

The first norm algorithm is less complex compared with the second norm algorithm. It is powerful 

when the training dataset has outliers. The first norm method should be used to ignore the outliers 

when the data is noisy [9,10]. 

The primal Lagrangian for first norm problem above is: 

𝐿𝑝(𝑤, 𝑏, 𝜉, 𝛼, ϓ) =
1

2
𝑤𝑇𝑤 + 𝑐 ∑ ϓ𝑖𝜉𝑖

𝑛

𝑖=1

−  ∑[𝑦𝑖(𝑤𝑇𝑥 + 𝑏) − 1 + 𝜉𝑖]

𝑛

𝑖=1

− ∑ ϓ𝑖𝜉𝑖

𝑛

𝑖=1

. 

with  𝛼𝑖 ≥ 0    𝑎𝑛𝑑   ϓ𝑖 ≥ 0. substituting 



IHJPAS. 37 (1) 2024 

418 
 

𝜕𝐿

𝜕𝑤
= 𝑤 − ∑ 𝑦𝑖𝛼𝑖𝑥𝑖

𝑛

𝑖=1

= 0;      
𝜕𝐿

𝜕𝜉
= 𝑐𝜉 − 𝛼 = 0;       

𝜕𝐿

𝜕𝑏
= ∑ 𝑦𝑖𝛼𝑖 = 0

𝑛

𝑖=1

. 

Into the initial Lagrangian, we get the dual problem: 

maximize 𝐿𝑑(𝛼, ϓ) = ∑ 𝛼𝑖
𝑛
𝑖=1 −

1

2
∑ ∑ 𝑦𝑖𝑦𝑗𝛼𝑖𝛼𝑗𝑥𝑗

𝑇𝑥𝑖 − ∑ 𝛼𝑖𝜉𝑖 − ∑ ϓ𝑖𝜉𝑖 +𝑛
𝑖=1

𝑛
𝑖=1

𝑛
𝑗=1

𝑛
𝑖=1

𝑐 ∑ 𝜉𝑖 = ∑ 𝛼𝑖
𝑛
𝑖=1 −

1

2
∑ ∑ 𝑦𝑖𝑦𝑗𝛼𝑖𝛼𝑗𝑥𝑗

𝑇𝑥𝑖
𝑛
𝑖=1

𝑛
𝑖=1

𝑛
𝑖=1 . 

subject to 0 ≤ 𝛼𝑖 ≤ 𝑐,   ∑ 𝛼𝑖𝑦𝑖
𝑛
𝑖=1 = 0. 

Remarkably the dual problem objective function is the same as that of the linearly separable 

case that was discussed previously. This is because the cancellation depends on 𝑐 = 𝛼𝑖 + 𝑦𝑖. Now, 

when 𝛼𝑖 ≥ 0 and ϓ𝑖 ≥ 0, we get 0 ≤ 𝛼𝑖 ≤ 𝑐. When we solve the Quadratic Programming (QP) 

problem for 𝛼𝑖 , the following optimal decision plane with the margin can be gotten: 

(∑ ∑ 𝛼𝑖𝛼𝑗𝑦𝑖𝑦𝑗𝑥𝑖
𝑇𝑥𝑗

𝑗∈𝑆𝑖∈𝑆
)−1/2 

 

4. Kernels Transformation 

The kernel transformation techniques can be applied when, the condition set of Karush-Kuhn-

Tucker (KKT) are satisfied. Eq (1) is linear  in terms of the new space that ɸ (x) maps the data to 

non-linear in the space, see Figure 3. The most common kernels are: linear, polynomial, sigmoid 

or Multi-Layer Perceptron (MLP) and Gaussian or Radial Basis Function (RPF) [11-13]. Their 

expressions are as follows: 

Linear kernel:𝐾(𝑥𝑖 , 𝑥𝑗) = 𝑥𝑖
𝑇𝑥𝑗 . 

Polynomial kernel:𝐾(𝑥𝑖, 𝑥𝑗) = (1 + 𝑥𝑖
𝑇𝑥𝑗)𝑝. 

Sigmoid (MLP) kernel: 𝐾(𝑥𝑖, 𝑥𝑗) = 𝑡𝑎𝑛ℎ(𝑘1𝑥𝑖
𝑇𝑥𝑗 + 𝑘2). 

Gaussian (RBF) kernel: 𝐾(𝑥𝑖, 𝑥𝑗) = 𝑒𝑥𝑝[
−(𝑥𝑖−𝑥𝑗)𝑇(𝑥𝑖−𝑥𝑗)

2𝜎2 ] 

  

Figure 3. Tow-dimensional space verses three-dimensional space. 

 



IHJPAS. 37 (1) 2024 

419 
 

we define the kernel function as K(𝑥𝑖, 𝑥𝑗) =< ɸ(𝑥𝑖), ɸ(𝑥𝑗) >= ɸ(𝑥𝑖)𝑇ɸ(𝑥𝑗) where ɸ is a 

mapping from input space to output space, see Figure 4. 

 

  
(a) with x̅ = (x1, x2) ∈ R2 ito a feature space (b) with 𝑧̅ = (𝑧1, 𝑧2, 𝑧3) ∈ R3, 𝑢𝑠𝑖𝑛𝑔 : 𝑅2 →

𝑅3 
Figure 4. Transforming a nonlinear data set using a kernel. 

 

Now, the corresponding dual form is 

𝐿(𝛼) = ∑ 𝛼𝑖
𝑛
𝑖=1 −

1

2
∑ ∑ 𝛼𝑖𝛼𝑗𝑦𝑖𝑦𝑗𝐾(𝑥𝑖, 𝑥𝑗)𝑛

𝑗=1
𝑛
𝑖=1 . 

Subject to ∑ 𝛼𝑖𝑦𝑖
𝑛
𝑖=1 = 0      𝛼𝑖 ≥ 0,     𝑖 = 1, . . . , 𝑛. 

   (4) 

Eq (4) is called a cost function, and it is convex and quadratic in terms of the unknown 

parameters. It can be solved using quadratic programming. The final decision rule for 

classification using KKT conditions is [14,15]: 

𝐿(𝑥, 𝛼∗, 𝛽0) = ∑ 𝑦𝑖𝛼𝑖
∗

𝑁𝑆

𝑖=1
𝐾(𝑥𝑖, 𝑥) + 𝛽0. 

Where 𝑁𝑆 is the number of support vectors, and 𝛼𝑖  the non-zero Lagrange multipliers that 

associated with the support vectors. 

 

5. Enhanced Stochastic Gradient Descent SVM 

The goal of this section is to minimize the following function, 

𝐿(𝛽) =
1

2
𝛽0

𝑇𝛽0 + 𝐾 ∑ max( 0, 1 − 𝑦𝑖𝛽
𝑇𝑥𝑖  )

𝑖

    (5) 

Equation(5) is a quadratic optimization problem and it is convex in 𝑝. In the previous section, 

the QP technique was used, but it is very slow. If there are no constraints, gradient descent can be 

used [16,17]. In general, the gradient goes in the opposite direction to get to the minimum, as 

shown in Figure 5.a, because the function is in the direction of the steepest slope. 

 



IHJPAS. 37 (1) 2024 

420 
 

  

a. Minimizing 𝐿(𝛽) using a general strategy b. Minimizing 𝐿(𝛽) using SGD 

Figure 5.Minimizing 𝐿(𝛽) using stochastic gradient descent. 

The general gradient descent SVM strategy (GD-SVM) for minimizing Eq.(5) starts with an 

initial value for 𝛽, say 𝛽0, then iterate until convergence. GD-SVM is faster than QP, but it is still 

slow because computing ∇(𝛽𝑗) takes 𝑝(𝑝) time in complexity, where 𝑝 is the size of the training 

dataset. If 𝑝 is large, GD-SVM is slow [18,19]. In Stochastic Gradient Descent (SGD), the value 

of the objective function is improved at each step. Evaluating the gradient for each training sample 

instead of evaluating it for all samples speeds up the process. This pressure is called Enhanced 

stochastic gradient descent SVM (ESGD-SVM). As can be seen in Fig.5.b, ESGD-SVM requires 

many more steps than the GD-SVM method, but it is less computationally intensive at each update; 

and ESGD-SVM is faster than the GD-SVM method. The ESGD-SVM algorithm (Algorithm 1) 

is guaranteed to converge to the minimum of 𝑝 if ,𝑝-𝑝. is small enough [20-22]. 

 

Algorithm 1: SVM using Stochastic Gradient Descent (ESGD-SVM) 

Given a training set 𝑆 = {(𝑥𝑖, 𝑦𝑖): 𝑥 ∈ ℝ𝑛 𝑎𝑛𝑑 𝑦 ∈ {−1, +1}} 

Repeat until convergence 

1. Initial value: 𝛽0. 

2. For 𝑖 = 1, … , k: 

     i. compute: ∇(𝛽𝑗) =  
𝜕𝑓(𝛽,𝛽0)

𝜕𝛽j
 = 𝛽j + R ∑

𝜕𝐿(𝑥𝑖,𝑦𝑖)

𝜕𝛽j
𝑛
𝑖=1 , where 𝑅 is a regularization factor.  

     ii. recompute 𝛽 as follows: 𝛽𝑗  ←  𝛽𝑗 − 𝜂∇(𝛽𝑗), where 𝜂 is the learning rate value. 

     iii. ∇Jt(𝛽𝑡) =  
1

2
𝛽0

𝑇𝛽0 + 𝐾 · 𝑁 · max (0,1 − 𝑦𝑖𝛽
𝑇𝑥𝑖), where 𝑁 is the number of training 

examples. 

     iv. Update 𝛽 as follows: 𝛽𝑡 ← 𝛽𝑡−1 − 𝛾𝑡∇Jt(𝛽𝑡−1). 

3. Repeat (𝑥𝑖, 𝑦𝑖) to make a full dataset and take the derivative of the SVM objective at the 

current 𝛽𝑡−1 to be ∇Jt(𝛽𝑡−1). 

4. Return final 𝛽. 

 

 



IHJPAS. 37 (1) 2024 

421 
 

6. Simulation Studies 

To test the GD-SVM and ESGD-SVM methods, two simulation datasets with 100 and 200 

observations are created. Each dataset contains a set of variables and a response comprising two 

classes. In real life, it is not known whether the data sets are linearly or non-linearly separable, so 

complex data sets (non-linearly separable) are generated. GD-SVM and ESGD-SVM are applied 

to the two data sets by using different types of kernels. In both methods, the MLP kernel provides 

the best accuracy in the two datasets [23,24]. In Table 1, GD-SVM and ESGD-SVM are compared 

in terms of the best value of 𝑝, the number of support vectors, using sensitivity (also called true-

positive rate) and specificity (also called true-negative rate) [25,26]. 

Table 1. Comparison between GD-SVM & ESGD-SVM for a data set with 100 and 200 observations. 

Sample size Method 
Kernel 

Type 

Best 𝑲 

Value 

Number of 

Support 

Vectors 

Sensitivity% Specificity% Accuracy 

100 

Observations 

GD-SVM 

Linear 0.78 13 82.30 89.43 86.43 

Polynomial 0.45 31 94.40 92.58 93.56 

RBF 0.32 19 92.74 94.01 93.01 

MLP 0.23 22 90.95 94.33 92.33 

ESGD-

SVM 

Linear 0.96 12 92.30 97.43 93.45 

Polynomial 0.55 19 95.40 96.53 95.53 

RBF 0.37 15 96.74 97.01 96.53 

MLP 0.33 16 98.95 96.33 97.33 

200 

Observations  

GD-

SVM 

Linear 0.83 23 81.31 87.43 85.43 

Polynomial 0.26 41 89.44 91.55 90.54 

RBF 0.39 28 89.74 84.01 87.01 

MLP 0.24 33 89.94 91.73 90.72 

ESGD-

SVM 

Linear 0.92 14 80.33 83.83 82.81 

Polynomial 0.57 29 87.43 88.58 86.54 

RBF 0.48 25 88.44 87.97 87.99 

MLP 0.37 28 86.41 85.96 85.66 

 

7. Real Dataset 

The modification of the SVM using stochastic gradient descent is the main topic of this paper. 

For real data applications, we have used South African heart disease data. The dataset was 

downloaded from [18]. In this dataset, a historical sample of men in the Western Cape of South 

Africa, a region with a high incidence of cardiovascular disease, is described. The following patient 

characteristics were recorded for each high-risk individual: Factors such as age, type A behavior, 

family history of heart disease, systolic blood pressure, cumulative cigarette consumption, low-

density lipoprotein cholesterol, body fat percentage and obesity. The total number of samples in 

this data collection is 462. Obesity refers to a high percentage of body fat, while obesity is defined 

by a high weight-to-height ratio (body mass index, BMI) [27] Excessive antagonism, aggression, 

and competitiveness are hallmarks of the Type A personality. We will see if we can extrapolate 

ldl from the available data. Since low-density lipoprotein (ldl) cholesterol is the "bad" cholesterol, 

elevated levels are thought to be associated with obesity and adiposity. The aim is to create a 

predictive model for ldl by selecting the most important factors [28]. 

In Table (2), we examine the Pearson correlation coefficient between the groups. It can be seen 

that there is a high significant correlation (marked with **), a significant correlation (marked with 

*) and a weak correlation between the variables [29,30]. 



IHJPAS. 37 (1) 2024 

422 
 

Table 2. Pearson correlation matrix between variables of heart disease dataset 

Pearson 

Correlation 
sbp tobacco ldl adiposity typea obesity alcohol 

tobacco 
R 

P 

0.212 

0.092 
      

ldl 
R 

P 

0.158 

0.958 

0.159 

0.203 
     

adiposity 
R 

P 

0.357* 

0.043 

0.287 

0.349 

0.440** 

0.002 
    

typea 
R 

P 

-0.057 

0.945 

-0.015 

0.503 

0.440* 

0.039 

-0.043 

0.203 
   

obesity 
R 

P 

0.238* 

0.043 

0.125 

0.349 

0.331* 

0.049 

0.717** 

0.014 

0.074 

0.249 
  

alcohol 
R 

P 

0.140 

0.4l3 

0.201 

0.459 

-0.033 

0.953 

0.123 

0.034 

0.039 

0.254 

0.052 

0.359 
 

age 
R 

P 

0.389* 

0.042 

0.450** 

0.011 

0.312 

0.059 

0.626** 

0.023 

-0.103 

0.539 

0.692** 

0.024 

0.101 

0.239 

R: Pearson correlation, P: p value 

**High significant correlation between variables ( p value < 0.01). 

*Significant correlation between variables (p value < 0.05). 

 

The estimated coefficient for the model was calculated in Table (3), some variables were highly 

significant since p value was less than 0.02, and some were significant where p value < 0.05. In 

general, 6 variables are selected as important variables in the dataset. 

 

 

 

 

 

 

 

 

Table 3. Estimated coefficient for model  using heart disease dataset. 

Variables in the 

Equation 
�̂� S.E. Wald P-value Exp(B) 

Constant -5.225 1.315 15.782 <0.001** 0.005 

sbp 0.007 0.006 1.288 0.256 1.007 

tobacco 0.379 0.027 8.903 0.003* 1.083 

ldl 0.174 0.06 8.498 0.004* 1.19 

adiposity 0.019 0.029 0.403 0.526 1.019 

famhist -0.925 0.228 16.488 <0.001** 0.396 

typea 0.340 0.012 10.329 0.001* 1.04 

obesity -0.063 0.044 2.021 0.155 0.939 

alcohol 0.002 0.004 0.001 0.978 1 

age 0.453 0.012 13.901 <0.001** 1.046 

*Significant effect for the parameter in the equation. 

**High significant effect for the parameter in the equation. 

 



IHJPAS. 37 (1) 2024 

423 
 

Instead of using 6 variables in the model as it was illustrated in Table (3), the principal component variable 

can be used  to reduce the components. In Figure 6.a, we draw a curve as a scree plot to show the number 

of components depends on their eigenvalues. Using an eigenvalue equal to 1 as a cut off, three components 

(RC1, RC2, RC3) are satisfactory to represent all the variables in the dataset. Figure 6.b shows the 

connection between the variables and their components. 

 

  

a. Number of principles that should be used in 

EDSGD SVM 

b. Diagram for connection between variables and 

their component. 

Figure 6. Importance variables for heart disease dataset using PCA technique. 

The summary of the main principles and their weight according to the variable that is used in the 

real dataset is shown in Table 4. 

 

 
 

Table 4. Principle components for the dataset. 

Component Loadings 

Variables RC1 RC2 RC3 

sbp    

tobacco  0.683  

ldl 0.747   

adiposity 0.868   

typea   0.959 

obesity 0.877   

alcohol  0.853  

age 0.477   

 

Figure 7 shows the important variables and their connections. In Figure 7.a the important 

variables are sorted in order. It shows that age and ldl are the most important variables followed 

by tobacco and typea. Figure 7.b shows the connection between variables. We see the strong 

connections are marked with thick blue lines, and the weak connection were marked with thin blue 

lines. 

 



IHJPAS. 37 (1) 2024 

424 
 

 
a. Sorting variables depend on their importance 

 
b. The connection between variables 

Figure 7. Path diagram for the relation between the variables and the component. 

 

8. Results and Discussion 

 The enhanced version of SVM (ESGD-SVM) is applied to the heart disease dataset. A comparison between 

some different kernel functions had been shown. In application, we applied the most common kernel which 

is linear, polynomial, RBF and MLP kernels. In addition, the proposed method was compared with some 

common methods. These methods used the same dataset for leukemia classification which are k-nearest 

neighbor random forest and naïve Bayes. In Figure10, ESGD-SVM classification method for the heart 

disease dataset is plotted. The two most important variables, which are tobacco and ldl are used for 

visualization. As it is shown from the plot (Figure 8) and the table (Table 5) the best version for ESGD-

SVM method is satisfied when the RBF kernel is applied. 

 The MLP kernel gets 98.10% accuracy which is the highest performance compared with other kernels. 

SGD-SVM performed 96.53% accuracy for the linear kernel, 98.03% accuracy for the polynomial kernel, 

and 96.53% accuracy for RBF kernel. ESGD-SVM performed much better than other methods where 

logistic regression performed 87.42% accuracy, k-nearest neighbor performed 86.70% accuracy, random 

forest performed 87.33% accuracy, and naïve Bayes performed 84.34% accuracy. 

 

 

 

 
a. Linear kernel 

 
b. Polynomial kernel 



IHJPAS. 37 (1) 2024 

425 
 

 
c. RBF kernel 

 
d. MLP kernel 

Figure 8. Classification for heart disease dataset using ESGD-SVM. 

 

Table 5. Comparison between ESGD-SVM, logistic regression, k-nearest neighbors, random forest, and    naive 

Bayes for classification heart disease dataset. 

Methods 
Number of 

Support Vectors 
Sensitivity % Specificity % Accuracy Rate % 

ESGD-SVM 

Linear kernel 42 96.61  98.67 96.53 

Polynomial kernel 96 98.19 94.19 98.03 

RBF kernel 51 96.61 98.67 96.53 

MLP kernel 48 97.21 99.10 98.10 

Logistic Regression 86.39 89.48 87.42 

k-nearest neighbors 85.20 88.40 86.70 

random forest 89.33 84.87 87.33 

naive Bayes 86.49 82.71 84.34 

 

The Receiver Operating Characteristic curve, or ROC curve, is shown. ROC is a graphical plot 

that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is 

varied for both the Logistic regression model and ESGD-SVM models. 

 

 

  

a. ROC for ESGD-SVM model b. ROC for Logistic regression model 

Figure 9. ROC curve for both ESGD-SVM and Logistic regression models. 



IHJPAS. 37 (1) 2024 

426 
 

9. Conclusion 

This paper presents the ESGD-SVM method. The stochastic gradient descent process is used to 

develop the method. The kernel transformation technique and dimensionality reduction for 

variables are used to achieve the best classification accuracy with ESGD-SVM. Two simulation 

datasets are used to test the implementation of the method. The results show that ESGD-SVM has 

higher accuracy compared to some other classification methods: logistic regression, k-nearest 

neighbors, random forest and naive Bayes. When the method was applied to a real dataset (heart 

disease), it was found that the highest accuracy (98.10%) was achieved by applying the MLP 

kernel. 

Acknowledgments 

The authors are very grateful to the reviewers for their valuable comments and suggestions to improve the 

paper 

Conflict of Interest 

The authors declare that they have no conflicts of interest. 

Funding 

There is no financial support in preparation for the publication. 

 

References 

1.  Zou, X.; Hu, Y.; Tian, Z.; Shen, K. Logistic regression model optimization and case analysis. 

IEEE 7th international conference on computer science and network technology (ICCSNT) 

2019, 7, 135-139. 

2. Liaw, A.; Wiener, M. Classification and regression by random Forest. R news. 2002, 3,18-22. 

3. Khorshid, S.F.; Abdulazeez, A.M. Breast cancer diagnosis based on k-nearest neighbors: a 

review. PalArch's Journal of Archaeology of Egypt/Egyptology. 2021, 18, 1927-51. 

4. Chen, S.; Webb, G.I.; Liu, L.; Ma, X. A. Novel selective naïve Bayes algorithm. Knowledge-

Based Systems. 2020, 192, 105361 

5. Choubey, D.K.; Kumar, M.; Shukla, V.; Tripathi, S.; Dhandhania, V.K.; Comparative analysis 

of classification methods with PCA and LDA for diabetes. Current diabetes reviews. 2020, 16, 

833-50.  

6. Tawfiq, L.N.; Rashid, T.A. On Comparison Between Radial Basis Function and Wavelet Basis 

Functions Neural Networks. Ibn AL-Haitham Journal For Pure and Applied Science. 2017, 

23, 184-92. 

7. Zhi, J.; Sun, J.; Wang, Z.; Ding, W. Support vector machine classifier for prediction of the 

metastasis of colorectal cancer. Int J Mol Med. 2018, 41, 1419-26. 

8. Cervantes, J.; Garcia-Lamont, F.; Rodríguez-Mazahua, L.; Lopez, A. A comprehensive survey 

on support vector machine classification: Applications, challenges and trends. 

Neurocomputing. 2020, 408, 189-215. 

9. Hekmatmanesh, A.; Wu, H.; Jamaloo, F.; Li, M.; Handroos, H. A combination of CSP-based 

method with soft margin SVM classifier and generalized RBF kernel for imagery-based brain 

computer interface applications. Multimedia Tools and Applications. 2020, 79, 17521-49. 

10. Wang, Y.; Yu, W.; Fang, Z. Multiple kernel-based SVM classification of hyperspectral images 

by combining spectral, spatial, and semantic information. Remote Sensing. 2020, 12, 120. 



IHJPAS. 37 (1) 2024 

427 
 

11. RAHEEM, S. H; KALAF, B. A.; SALMAN, A. N. Comparison of Some of Estimation 

methods of Stress-Strength Model: R= P (Y< X< Z). Baghdad Science Journal, 2021, 18.2, 

1103-1103.  

12. JEBUR, I. G.; KALAF, B. A.; SALMAN, A. N. An efficient shrinkage estimators for 

generalized inverse rayleigh distribution based on bounded and series stress-strength models. 

In: Journal of Physics: Conference Series. IOP Publishing, 2021, 012054. 

13. Mahdi, G.J.; Mohammed, N.J.; Al-Sharea, Z.I. Regression shrinkage and selection variables 

via an adaptive elastic net model. In Journal of Physics: Conference Series 2021, 1879, 032014. 

14. Qingyang, Z.; Ghadeer, M.; Jian, T.; Hao, C.; A graph-based multi-sample test for identifying 

pathways associated with cancer progression. Computational Biology and Chemistry, 2020, 

87: 107285. 

15. ZHANG, Q.; DAO, T. A distance based multisampling test for high-dimensional 

compositional data with applications to the human microbiome. BMC bioinformatics, 2020, 

21, 1-17. 

16. Mahdi, G.J, Kalaf,  B.A.; Khaleel, M.A. Enhanced supervised principal component analysis 

for cancer classification. Iraqi Journal of Science. 2021, 1321-33.  

17. Mseer, H.A.; Mahdi, G.J. Comparison among variable selection models and its application to 

health dataset. InAIP Conference Proceedings 2023, 1, 2414.  

18. Jabbar, A.K. New transform Fundamental properties and its applications. Ibn Al-Haitham 

Journal for Pure and Applied Sciences. 2018, 31, 1-10. 

19. Mahdi, G.J.; A Modified Support Vector Machine Classifiers Using Stochastic Gradient 

Descent with Application to Leukemia Cancer Type Dataset. Baghdad Science Journal. 

2020,17,1255-69.  

20. Raheem, S.H.; Kalaf, B.A.; Salman A.N. Comparison of Some of Estimation methods of 

Stress-Strength Model: R= P (Y< X< Z). Baghdad Science Journal. 2021,18,1103-17.  

21. Salah, O.M.; Mahdi, G.J.; Al-Latif, I.A. A modified ARIMA model for forecasting chemical 

sales in the USA. In Journal of Physics: Conference Series 2021, 1879, 032008. 

22. AL-NOOR, N. H.; KHALEEL, M. A.; MOHAMMED, G. J. Theory and applications of 

Marshall Olkin Marshall Olkin Weibull distribution. In: Journal of Physics: Conference 

Series. 2021,20, 012101. 

23. SHEAH, R. H.; ABBAS, I. T. Using multi-objective bat algorithm for solving multi-objective 

non-linear programming problem. Iraqi Journal of Science, 2021, 997-1015. 

24. MOHAMMED, M. J.; MOHAMMED, A. T. Analysis of an Agriculture Data Using Markov 

Basis for Independent Model. In: Journal of Physics: Conference Series. 2020, 012071. 

25. MOHAMMED, M. J.; MOHAMMED, A. T. Parameter estimation of inverse exponential 

Rayleigh distribution based on classical methods. International Journal of Nonlinear Analysis 

and Applications, 2021, 12, 935-944. 

26. Bartley, C. Replication Data for: South African Heart Disease" Available online: 

https://doi.org/10.7910/DVN/76SIQD  Harvard Dataverse, V1, 2016. 

27. Bayda, A. Abdul Jabbar, K. B.; Iraq, T. A. Mohd, R. A.; Lee, L. S. Application of simulated 

annealing to solve multi-objectives for aggregate production planning. In: AIP Conference 

Proceedings.  2016, 1739, 020086. 

28. Bogatinovski, J.; Ljupčo, T.; Sašo, D.; Dragi, Kocev. Comprehensive comparative study of 

multi-label classification methods. Expert Systems with Applications. 2022, 203, 117215. 

https://doi.org/10.7910/DVN/76SIQD
https://doi.org/10.7910/DVN/76SIQD


IHJPAS. 37 (1) 2024 

428 
 

29. FJELLSTRÖM, C.; NYSTRÖM, Kaj. Deep learning, stochastic gradient descent and diffusion 

maps. Journal of Computational Mathematics and Data Science. 2022, 4, 100054.  

30. HASSAN, A. S.; KHALEEL, M. A.; MOHAMD, R. E. An extension of exponentiated Lomax 

distribution with application to lifetime data. Thailand Statistician. 2021, 19, 484-500.