RUHUNA JOURNAL OF SCIENCE 
Vol 13 (1): 14-28, June 2022 
eISSN: 2536-8400                                                 Faculty of Science 
http://doi.org/10.4038/rjs.v13i1.112                                                                                   University of Ruhuna 

Faculty of Science, University of Ruhuna    

Sri Lanka 
14 

Generalized Stochastic Restricted LARS Algorithm  

Manickavasagar Kayanan1,2 and Pushpakanthie Wijekoon3  

1Postgraduate Institute of Science, University of Peradeniya, Peradeniya, Sri Lanka 
2 Department of Physical Science, University of Vavuniya, Vavuniya, Sri Lanka 
3 Department of Statistics and Computer Science, University of Peradeniya, Peradeniya, Sri Lanka 

 

*Correspondence: mgayanan@vau.jfn.ac.lk;  ORCID: https://orcid.org/0000-0003-2662-4383 

 

Received: 29th July 2021; Revised: 21st April 2022; Accepted: 17th May 2022 

Abstract The Least Absolute Shrinkage and Selection Operator (LASSO) is used 

to tackle both the multicollinearity issue and the variable selection concurrently 

in the linear regression model. The Least Angle Regression (LARS) algorithm has 

been used widely to produce LASSO solutions. However, this algorithm is 

unreliable when high multicollinearity exists among regressor variables. One 

solution to improve the estimation of regression parameters when 

multicollinearity exists is adding preliminary information about the regression 

coefficient to the model as either exact linear restrictions or stochastic linear 

restrictions. Based on this solution, this article proposed a generalized version of 

the stochastic restricted LARS algorithm, which combines LASSO with existing 

stochastic restricted estimators. Further, we examined the performance of the 

proposed algorithm by employing a Monte Carlo simulation study and a 

numerical example. 

Keywords: LASSO, LARS, Stochastic Linear Restrictions.  

1   Introduction 

The biased estimators such as Ridge Estimator (RE) (Hoerl and Kennard 1970), 

Almost Unbiased Ridge Estimator (AURE) (Singh et al. 1986), Liu Estimator (LE) 

(Liu 1993), Almost Unbiased Liu Estimator (AULE) (Akdeniz and Kaçiranlar 1995), 

Principle Component Regression Estimator (PCRE) (Massy 1965), r-k class estimator 

(Baye and Parker 1984), r-d class estimator (Kaçiranlar and Sakallıoğlu 2001) and 

Sample Information Optimal Estimator (SIOE) (Kayanan and Wijekoon 2019) have 

been widely used in literature to resolve multicollinearity issue in the linear regression 

model. However, these estimators yield high bias when the number of explanatory 

variables is high, and they do not consider about irrelevant variables while fitting 

models. For high dimensional data, having many variables in the model and 

multicollinearity are major issues. To tackle these matters, Tibshirani (1996) 

introduced Least Absolute Shrinkage and Selection Operator (LASSO). The LASSO 

https://rjs.ruh.ac.lk/index.php/rjs/index
https://creativecommons.org/licenses/by-nc/4.0/
mailto:mgayanan@vau.jfn.ac.lk
https://orcid.org/0000-0003-2662-4383
https://orcid.org/0000-0003-2662-4383


 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
15 

is a shrinkage method that was originally used for regularization and variable selection 

in the linear regression model. The Least Angle Regression (LARS) (Efron et al. 2004) 

algorithm has been used to obtain the estimates of LASSO. Zou and Hastie (2005) have 

shown that the LASSO is unsteady when severe multicollinearity exists between the 

explanatory variables. Therefore, they suggested Elastic Net (ENet) estimator by 

combining LASSO and RE as a solution for this issue. Furthermore, they proposed 

LARS-EN algorithm to attain ENet solutions, which is a modified version of the LARS 

algorithm. Besides, Kayanan and Wijekoon (2020b) proposed a generalized version of 

LARS (GLARS) algorithm to combine LASSO with RE and the other biased 

estimators based on sample information such as AURE, LE, AULE, PCRE, r-k class 

estimator, and r-d class estimator. Finally, they have shown that the GLARS algorithm 

performs well when it combines LASSO with r-k class and r-d class estimators. 

According to literature, the parameter estimation can be strengthened if prior 

knowledge about the regression coefficient is applied. The prior information on 

regression coefficients can be defined in the form of exact linear restrictions or 

stochastic linear restrictions. Many researchers proposed stochastic restricted 

estimators such as Mixed Regression Estimator (MRE) (Theil and Goldberger 1961) 

Stochastic Restricted Ridge Estimator (SRRE) (Li and Yang 2010), Stochastic 

Restricted Almost Unbiased Ridge Estimator (SRAURE) (Jibo and Hu 2014), 

Stochastic Restricted Liu Estimator (SRLE) (Hubert and Wijekoon 2006), Stochastic 

Restricted Almost Unbiased Liu Estimator (SRAULE) (Jibo and Hu 2014), Stochastic 

Restricted Principle Component Regression Estimator (SRPCRE) (He and Wu 2014), 

Stochastic Restricted r-k class estimator (SRrk) (Jibo 2014), Stochastic Restricted r-d 

class estimator (SRrd) (Jibo 2014), and Stochastic Restricted Optimal Estimator 

(SROE) (Kayanan and Wijekoon 2019) to incorporate prior information to the 

regression coefficient. Stochastic restricted estimators also have the same issue as 

biased estimators when the linear regression model contains numerous predictors. To 

handle this problem, Kayanan and Wijekoon (2020a) proposed a stochastic restricted 

LARS (SRLARS) algorithm to combine LASSO and MRE, and showed the superiority 

of the SRLARS over LARS algorithm. 

This article proposes a generalized version of the stochastic restricted LARS 

algorithm, namely SRGLARS, to combine LASSO with other stochastic restricted 

estimators. The prediction performance of the SRGLARS algorithm was examined by 

employing a Monte-Carlo simulation and using a real-world example in the Root Mean 

Square Error (RMSE) criterion.  

2 Model Specification and the Estimators  

Consider the linear regression model   

𝒚 = 𝑿𝜷 + 𝜺,     (2.1) 
where 𝑿 is 𝑛 × 𝑝 matrix of explanatory variables, 𝜷 be the 𝑝 × 1 vector of unknown 

coefficients, and 𝜺 be the 𝑛 × 1 vector of disturbances such that 𝜺 ∼ 𝑁 (0,𝜎2 𝑰). 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
16 

Assume that there exists prior information on β, which may be expressed as a 

stochastic linear restriction, as (Theil & Goldberger, 1961)  

𝝋 =  𝑹𝜷 +  𝒗,    (2.2) 
where φ be the 𝑞 × 1 vector, 𝑹 be the 𝑞 × 𝑝 matrix with rank 𝑞, 𝒗 be the 𝑞 × 1 

vector of disturbances, such that 𝒗 ∼ 𝑁 (0,𝜎2𝑾 ), 𝑾 is positive definite, and 
𝐸(𝒗𝜺′) =  0. 

To make the variable selection and handle multicollinearity issue by incorporating 

prior information defined in model (2.2), Kayanan and Wijekoon (2020a) suggested 

Stochastic Restricted LASSO Type Estimator (SRLASSO) for model (2.1) as 

�̂�𝑆𝑅𝐿𝐴𝑆𝑆𝑂 = argmin
𝜷

{(𝒚 − 𝑿𝜷)′(𝒚 − 𝑿𝜷)} 

subject to ∑ |𝛽𝑗|
𝑝
𝑗=1 ≤ 𝑡 and 𝑹𝜷 = 𝝋 − 𝒗,                                              (2.3) 

where 𝑡 > 0 is a turning parameter. 
 

Further, Kayanan and Wijekoon (2020a) proposed Stochastic Restricted LARS 

(SRLARS) algorithm to find the SRLASSO estimates. Note that SRLARS combines 

LASSO and MRE to find the estimates. To improve the SRLARS solutions, this article 

proposes a generalized version of SRLARS (SRGLARS) to combine LASSO with 

other stochastic restricted estimators such as SRRE, SRAURE, SRLE, SRAULE, 

SRPCRE, SRrk, SRrd and SROE. 

Kayanan and Wijekoon (2018, 2019) proposed a generalized form to express the 

estimators MRE, SRRE, SRAURE, SRLE, SRAULE, SRPCRE, SRrk, SRrd, and 

SROE as 

�̂�𝐺 = 𝑮(𝑿
′𝑿 + 𝑹′𝑾−𝟏𝑹)−𝟏 (𝑿′𝒚 + 𝑹′𝑾−𝟏𝝋), (2.4) 

 where  

�̂�𝐺 =

{
 
 
 
 
 

 
 
 
 
 
�̂�𝑀𝑅𝐸            if   𝑮 = 𝑰                                                 

�̂�𝑆𝑅𝑅𝐸           if   𝑮 = (𝑿
′𝑿 + 𝑘𝑰)−1𝑿′𝑿                   

�̂�𝑆𝑅𝐴𝑈𝑅𝐸       if   𝑮 = (𝑰 − 𝑘
2(𝑿′𝑿 + 𝑘𝑰)−2)          

�̂�𝑆𝑅𝐿𝐸            if   𝑮 = (𝑿
′𝑿 + 𝑰)−1(𝑿′𝑿 + 𝑑𝑰)        

�̂�𝑆𝑅𝐴𝑈𝐿𝐸        if   𝑮 = (𝑰 − (1 − 𝑑)
2(𝑿′𝑿 + 𝑰)−2)  

�̂�𝑆𝑅𝑃𝐶𝑅𝐸        if   𝑮 = 𝑻ℎ𝑻ℎ
′                                             

�̂�𝑆𝑅𝑟𝑘            if   𝑮 = 𝑻ℎ𝑻ℎ
′ (𝑿′𝑿 + 𝑘𝑰)−1𝑿′𝑿          

�̂�𝑆𝑅𝑟𝑑            if   𝑮 = 𝑻ℎ𝑻ℎ
′ (𝑿′𝑿 + 𝑰)−1(𝑿′𝑿 + 𝑑𝑰)

�̂�𝑆𝑅𝑂𝐸            if   𝑮 = 𝜷∗𝜷∗
′(𝜎2(𝑿′𝑿)−𝟏 + 𝜷∗𝜷∗

′)
−1
 

 

 

Note that 𝑘 > 0 and 0 < 𝑑 < 1 are the shrinkage/regularization parameters, 𝑰 is the 
𝑝 × 𝑝 identity matrix, 𝑻ℎ = (𝑡1, 𝑡2. . . 𝑡ℎ)′ is the first ℎ columns of the eigenvectors of 
𝑿′𝑿, and 𝜷∗   is the normalized eigenvector corresponding to the largest eigenvalue of 
𝑿′𝑿. 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
17 

3   SRGLARS Algorithm 

Based on Kayanan and Wijekoon (2021a) and equation (2.4), the SRGLARS algorithm 

for model (2.1) is outlined below: 

  Algorithm 1: SRGLARS  

1: Standardize 𝑿 to have a mean zero with a standard deviation of one, and center the 
𝒚 to have a mean zero.   

2: Start with all estimates of the coefficients �̂� = 0 with the residuals 𝒓 = �̂� and 𝝉 =
�̂�. 

3: Find the predictor 𝑿𝒋 most correlated with 𝒓; 𝑗 =  1,2, . . . ,𝑝. 

4: Move the estimate of �̂�𝑗 from 0 towards the �̂�𝐺 direction until some other predictor 

𝑿𝒌 has as large a correlation with the current residual as 𝑿𝒋 does. 

5: Move �̂�𝑗 and �̂�𝑘 in the direction defined by their joint �̂�𝐺 direction of the current 

residual on (𝑿𝒋,𝑿𝒌), until some other predictor 𝑿𝒍 eventually earns its way into the 

active set. 

6: If a non-zero coefficient hits zero, drop its variable from the active set of variables 

and recomputed the current joint �̂�𝐺 direction. 
7: Repeat the steps 5 and 6 until SRGLARS conditions attained. 

 

The mathematical details of the SRLARS algorithm are as follows: 

Let us assume that the estimates of the coefficients �̂� and residuals 𝒓 and 𝝉 are 

(�̂�)
0
= [(𝛽1)0, (𝛽2)0,…,(𝛽𝑝)0

]
′
= 𝟎, 𝒓0  =  𝒚 and 𝝉0   =  𝝋. 

Find the predictor (𝑋𝑗)1 most correlated with 𝒓0.  

(𝑋𝑗)1 = max
𝑗
|𝐶𝑜𝑟(𝑋𝑗,𝒓0)|; 𝑗 ∈ (1,2,…,𝑝)    (3.1) 

Then, increase the estimate of respective regression coefficient (𝛽𝑗)1
 from 0 unto 

any other predictor (𝑋𝑗)2 has a high correlation with 𝒓1 as (𝑋𝑗)1 does. At this stage, 

SRGLARS moves in the equiangular direction between (𝑋𝑗)1 and (𝑋𝑗)2 rather than 

proceeding the path based on (𝑋𝑗)1.  

Similarly, in 𝑖𝑡ℎ run, the variable (𝑋𝑗)1 eventually acquires its path in the active set, 

and then SRGLARS moves in the equiangular direction among (𝑿)𝑖 =

[(𝑋𝑗)1
, (𝑋𝑗)2

,…,(𝑋𝑗)𝑖
]
′
.  Proceed to add variables to the active set in this way, running 

in the path established by the least angle direction. During this process, (�̂�)
𝑖
=

[(𝛽𝑗)1
,(𝛽𝑗)2

,…,(𝛽𝑗)𝑖
]
′
 is updating using the following formula: 

(�̂�)
𝑖
= (�̂�)

𝑖−1
+ 𝛼𝑖𝒖𝑖,   (3.2) 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
18 

where 𝛼𝑖 ∈ [0,1] which signifies how long the estimate runs in the path before 
another predictor enters the model and the path turns anew, and 𝒖𝑖 is the equiangular 
vector. 

The path vector 𝒖𝑖 is computed using the formula given below based on the 
generalized form defined in equation (2.4): 

𝒖𝑖 = 𝑮𝐸((𝑬)𝒊
′(𝑿′𝑿 + 𝑹′𝑾−𝟏𝑹)(𝑬)𝒊) 

−𝟏(𝑬)𝒊
′(𝑿′𝒓𝒊−𝟏 + 𝑹′𝑾

−𝟏𝝉𝒊−𝟏), (3.3) 

where (𝑬)𝒊 = [(𝒆𝑗)1
,(𝒆𝑗)2

,… ,(𝒆𝑗)𝑖
],  and  (𝑒𝑗)𝑖 be  the  𝑗

th  standard  unit  vector  

in 𝑅𝑝, which has the record of selected variables in every succeeding steps, and 𝑮𝐸 is 
a generalized matrix that can be changed by respective expressions for any of 

stochastic restricted estimators of our interest as outlined in Table 1. 

Then, 𝛼𝑖 be calculated as follows: 
𝛼𝑖 = min

 
{𝛼𝑖

+,𝛼𝑖
−,𝛼𝑖

∗},      (3.4)  

where 

𝛼𝑖
± =

𝐶𝑜𝑟((𝑋𝒋)𝑖
,𝒓𝒊−𝟏)±𝐶𝑜𝑟(𝑋𝑗,𝒓𝒊−𝟏)

𝐶𝑜𝑟((𝑋𝒋)𝑖
,𝒓𝒊−𝟏)±𝐶𝑜𝑟(𝑋𝑗,(𝑿)𝑖𝒖𝑖)

    (3.5) 

for any 𝑗  such that (𝛽𝑗)𝑖−1
= 0, and 

𝛼𝑖
∗ = −

(�̂�)
𝑖−1

𝒖𝑖
      (3.6) 

for any 𝑗 such that (𝛽𝑗)𝑖−1
≠ 0. 

If 𝛼𝑖 = 𝛼𝑖
∗, then (𝑬)𝒊 is reformed by deleting the column 𝑒𝑗 from (𝑬)𝒊−𝟏.  Then 𝒓𝒊 

and 𝝉𝒊 can be calculated as 
𝒓𝒊 = 𝒓𝒊−𝟏 − 𝛼𝑖(𝑿)𝑖𝒖𝑖   and     (3.7) 
𝝉𝒊 = 𝝉𝒊−𝟏 − 𝛼𝑖(𝑹)𝑖𝒖𝑖,     (3.8) 

where (𝑹)𝑖 = [(𝑹𝒋)1
,(𝑹𝒋)2

,…,(𝑹𝒋)𝑖
]. Then proceed to the next step where (𝑗)𝑖+1 

is the value of 𝑗 such that 𝛼𝑖 = 𝛼𝑖
+or 𝛼𝑖 = 𝛼𝑖

− or 𝛼𝑖 = 𝛼𝑖
∗ . Proceed with the algorithm 

until𝛼𝑖 = 1. 
 
Table 1: 𝐺𝐸 of the estimators for SRGLARS. 

Estimators                      𝑮𝐸 
MRE (𝑬)𝒊 
SRRE (𝑬)𝒊((𝑬)𝒊

′ (𝑿′𝑿 + 𝑘𝑰)(𝑬)𝒊)
−1(𝑬)𝒊

′ 𝑿′𝑿(𝑬)𝒊 
SRAURE (𝑬)𝒊(𝑰𝑃𝐸 − 𝑘

2((𝑬)𝒊
′ (𝑿′𝑿 + 𝑘𝑰)(𝑬)𝒊)

−2) 
SRLE (𝑬)𝒊((𝑬)𝒊

′ (𝑿′𝑿 + 𝑰)(𝑬)𝒊)
−1(𝑬)𝒊

′ (𝑿′𝑿 + 𝑑𝑰)(𝑬)𝒊 
SRAULE (𝑬)𝒊(𝑰𝑃𝐸 − (1 − 𝑑)

2((𝑬)𝒊
′ (𝑿′𝑿 + 𝑰)(𝑬)𝒊)

−2) 
SRPCRE 𝑻ℎ𝐸𝑻ℎ𝐸

′ (𝑬)𝒊 
SRrk 𝑻ℎ𝐸𝑻ℎ𝐸

′ (𝑬)𝒊((𝑬)𝒊
′ (𝑿′𝑿 + 𝑘𝑰)(𝑬)𝒊)

−1(𝑬)𝒊
′ 𝑿′𝑿(𝑬)𝒊 

SRrd 𝑻ℎ𝐸𝑻ℎ𝐸
′ (𝑬)𝒊((𝑬)𝒊

′ (𝑿′𝑿 + 𝑰)(𝑬)𝒊)
−1(𝑬)𝒊

′ (𝑿′𝑿 + 𝑑𝑰)(𝑬)𝒊 
SROE (𝑬)𝒊((𝑬)𝒊

′ 𝜷∗𝜷∗
′(𝑬)𝒊(𝜎

2((𝑬)𝒊
′ 𝑿′𝑿(𝑬)𝒊)

−𝟏 + (𝑬)𝒊
′ 𝜷∗𝜷∗

′(𝑬)𝒊)
−1

 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
19 

In Table 1, 𝑰𝑃𝐸 is the 𝑝𝐸 × 𝑝𝐸 identity matrix, 𝑝𝐸 is the amount of selected variables 
in each succeeding step, and 𝑻ℎ𝐸 = (𝑡1, 𝑡2. . . 𝑡ℎ𝐸)  is the first ℎ𝐸 column of the 
standardized eigenvectors of (𝑬)𝒊

′ 𝑿′𝑿(𝑬)𝒊. 
We can apply SRGLARS to consolidate LASSO and any of the stochastic restricted 

estimators listed in Table 1. The suitable value of regularization parameter 𝑘 or 𝑑 of 
the proposed algorithms can be chosen by 10-fold cross-validation for every 𝑡 as 
outlined in Appendix C. 

We can get a separate algorithm for each stochastic restricted estimator by referring 

SRGLARS as LARS-MRE, LARS-SRRE, LARS-SRAURE, LARS-SRLE, LARS-

SRAULE, LARS-SRPCRE, LARS-SRrk, LARS-SRrd, and LARS-SROE when GE 

equals to the corresponding expression of MRE, SRRE, SRAURE, SRLE, SRAULE, 

SRPCRE, SRrk, SRrd, and SROE, respectively. 

4 Discussion 

The performance of SRGLARS algorithm by considering different combinations of 

LASSO and stochastic restricted estimators were examined using RMSE criterion, 

which is the expected prediction error of the algorithm for each estimator, and is 

defined as 

𝑅𝑀𝑆𝐸(�̂�) = √
1

𝑛∗
  (𝒚∗ − 𝑿∗�̂�)

′
(𝒚∗ − 𝑿∗�̂�),   (4.1) 

where (𝒚∗,𝑿∗) are the new observations, 𝑛∗ is the size of new observations, and �̂� is 
the estimated value of 𝜷 by the corresponding algorithm. Note that we cannot provide 
theoretical conditions for the superiority of particular algorithms in variable selection 

methods. Therefore, we use a Monte Carlo simulation and a real-world example to 

compare the SRGLARS algorithms. 

 

4.1 Simulation study 

 

We used the following formula to generate the explanatory variables based on 

McDonald and Galarneau (1975): 

𝑥𝑖,𝑗  = √(1 − 𝜌
2)𝑧𝑖,𝑗 + 𝜌𝑧𝑖,𝑚+1;    𝑖 =  1,2, . . . ,𝑛.     𝑗 =  1,2, . . . ,𝑚.   (4.2) 

where 𝑧𝑖,𝑗 is an independent standard normal pseudo-random number, and 𝜌 is the 

correlation among any two explanatory variables. 

In this study, we have used 100 observations with 20 explanatory variables, in which 

70 observations were used to fit the model and 30 observations were used to compute 

the RMSE.  

 

A dependent variable is generated by using the following equation, 

𝑦𝑖 = 𝛽1𝑥𝑖,1 + 𝛽2𝑥𝑖,2 + ⋯+ 𝛽20𝑥𝑖,20 + 𝜀𝑖,  (4.3) 

where 𝜀𝑖 is a normal pseudo-random number with 𝐸(𝜀𝑖) = 0 and 𝑉 (𝜀𝑖) = 𝜎
2 = 1. 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
20 

We choose 𝛽 = (𝛽1,𝛽2,…,𝛽20) as the normalized eigenvector corresponding to the 
largest eigenvalue of 𝑿′𝑿 for which 𝜷′𝜷 =  1 (McDonald and Galarneau 1975).  Prior 
information was defined based on Nagar and Kakwani’s (1964) approach, which is 

described in Appendix B. Further, we have assumed the first four elements of OLSE 

estimates as 𝒃 (see Appendix B). To study the effects of various degrees of 
multicollinearity on the data, we pick 𝜌 = (0.5, 0.7, 0.9), which signifies weak, 
moderate, and high multicollinearity, respectively.   Since the execution time for the 

algorithm simulation is long due to cross-validation, we simulated only 50 datasets. 

Future simulation studies will be implemented with a greater number of simulations 

using cluster programming. Figures 1-3 and Tables 2-4 show the cross-validated RMSE 

and the median cross-validated RMSE of the SRGLARS algorithms, respectively, for 

50 simulated data.  

 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

Fig 1. Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.5 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Fig 2. Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.7 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
21 

 

  

 

 

 

 

 

 

 

 

Fig 3. Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.9 

Table 2: Median Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.5. 

Algorithms RMSE (k, d) t Selected variables 

LARS-MRE 3.2805 – 6.1630 15 

LARS-SRRE 3.3531 0.3 6.1777 16 

LARS-SRAURE   3.2832 0.1 6.1631 16 

LARS-SRLE 3.3531 0.7 6.1773 16 

LARS-SRAULE   3.3012 0.99 6.1630 16 

LARS-SRPCRE    3.3070 – 6.4343 16 

LARS-SRrk 3.3122 0.1 6.9493 17 

LARS-SRrd 3.2976 0.9 6.9497 16 

LARS-SROE 3.0182 – 4.1102 18 

Table 3: Median Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.7. 

Algorithms RMSE (k, d) t Selected variables 

LARS-MRE 3.2041 – 6.8458 15 

LARS-SRRE 3.3575 0.1 7.5043 16 

LARS-SRAURE 3.3884 0.8 6.7785 16 

LARS-SRLE 3.3537 0.9 7.5049 17 

LARS-SRAULE 3.3886 0.2 6.7786 16 

LARS-SRPCRE 3.3135 – 7.7712 16 

LARS-SRrk 3.2525 1.0 8.5411 17 

LARS-SRrd 3.2534 0.5 8.1097 17 

LARS-SROE 3.0324 – 4.3864 18 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
22 

Table 4: Median Cross-validated RMSE values of the SRGLARS algorithms when ρ = 0.9. 

Algorithms RMSE (k, d) t Selected variables 

LARS-MRE 3.3391 – 11.6628 16 

LARS-SRRE 3.2722 0.1 11.5897 17 

LARS-SRAURE   3.3354 0.1 11.6606 16 

LARS-SRLE 3.3110 0.99 11.6628 17 

LARS-SRAULE   3.3107 0.99 11.6628 16 

LARS-SRPCRE    3.3782 – 12.2288 16 

LARS-SRrk 3.3531 0.7 12.9540 17 

LARS-SRrd 3.3276 0.3 13.0038 17 

LARS-SROE     2.9878 – 4.5654 18 

 

From Figure 1-3, we can observe that the LARS-SROE algorithm outperformed 

other SRGLARS algorithms in RMSE criterion under all degrees of multicollinearity. 

From Tables 2-4, we observe that the SROE included more variables than the other 

SRGLARS algorithms, although it has minimum RMSE. Therefore, in practical 

situations, if a researcher intends to reduce the number of variables from the model, 

they may consider other SRGLARS algorithms based on the interested variables and 

prediction performance using the plot of coefficient paths discussed in the real-world 

example. 

 

4.2 Real-world example 
  

As a real-world data set, we considered the Prostate Cancer Data (Stamey et al. 1989), 

which is used by Tibshirani (1996), Efron et al. (2004) and Zou and Hastie (2005) to 

study the performance of LASSO, LARS algorithm and Enet. The Prostate Cancer 

Data contains 97 observations and 8 predictors such as log cancer volume (lcavol), log 

prostate weight (lweight), age, log of the amount of benign prostatic hyperplasia (lbph), 

seminal vesicle invasion (svi), log capsular penetration (lcp), Gleason score (gleason) 

and percentage Gleason score 4 or 5 (pgg45). The dependent variable is the log of 

prostate-specific antigen (lpsa). The Variance Inflation Factor (VIF) values of the 

predictor variables are 3.09, 2.97, 2.47, 2.05, 1.95, 1.37, 1.36 and 1.32, and the 

condition number is 243, which exposes high multicollinearity between the predictor 

variables. This data set is associated in “lasso2” R package. We have used 67 

observations to fitting the model, and 30 observations to compute the RMSE. We have 

assumed (Nagar and Kakwani 1964) that the first three OLSE estimates of Prostate 

Cancer Data are unbiased, and we defined the prior information for this data based on 

Nagar and Kakwani’s (1964) approach, as described in Appendix B.  

The cross-validated RMSE of the SRGLARS algorithms are shown in Table 5, and 

coefficient paths of respective SRGLARS algorithm are shown in Figure 4 in Appendix 

A. 

 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
23 

Table 5: Cross-validated RMSE values of Prostate Cancer Data using SRGLARS. 

Algorithms RMSE (k, d) t Selected variables 

LARS-MRE 0.77784 – 1.5632 6 

LARS-SRRE 0.73567 0.38 1.7329 8 

LARS-SRAURE 0.77784 0.01 1.5632 6 

LARS-SRLE 0.73566 0.62 1.7331 8 

LARS-SRAULE 0.77784 0.99 1.5632 6 

LARS-SRPCRE 0.74084 – 1.5656 7 

LARS-SRrk 0.74083 0.04 1.5643 7 

LARS-SRrd 0.74084 0.96 1.5643 7 

LARS-SROE 0.69897 – 1.1226 8 

 

Table 5 shows that the LARS-SROE algorithm outperforms other algorithms on 

Prostate Cancer Data, which is agreed the results obtained in the simulation study. 

Further, we can note that LARS-SRRE, LARS-SRLE, and LARS-SROE did not make 

any variable selections on Prostate Cancer Data.  

Figure 4 (see Appendix A) shows that the choice of variables is different for each 

SRGLARS algorithms. Note that in all graphs, the order of selection of variables is the 

same except Figure 4(i). Further, the coefficients relevant to the variables lcp, age and 

gleason are very small for Fig 4((a)-(h)) and those coefficients are almost zero in Fig 

4(i). This indicates that the contribution from those three variables is very low for the 

fitted model.  

5   Conclusions  

This study presented a generalized version of stochastic restricted LARS (SRGLARS) 

algorithm to combine LASSO with existing stochastic restricted estimators. We have 

shown the superiority of SRGLARS when it combines LASSO with SROE (LARS-

SROE) using a Monte Carlo simulation and a real-world example in RMSE criterion. 

However, LARS-SROE is less likely to make variable selections when looking at the 

results directly. However, by examining the size of the coefficients as shown in the 

example, we can drop some irrelevant variables form the model. If a researcher wanted 

to reduce some variables from the model, the other combinations of the SRGLARS 

algorithm could be considered to fit the regression model by studying the RMSE and 

coefficients estimates based on the coefficients path. 

Acknowledgements 

Anonymous reviewers are acknowledged for their comments on the initial manuscript. 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
24 

References 

Akdeniz F, Kaçiranlar S. 1995. On the almost unbiased generalized liu estimator and unbiased estimation 

of the bias and mse. Communication in Statistics - Theory and Methods 24(7):1789–1797. 

Baye MR, Parker DF. 1984. Combining ridge and principal component regression: A money demand 

illustration. Communications in Statistics-Theory and Methods 13(2):197–205.  

Efron B, Hastie T, Johnstone I, Tibshirani R. 2004. Least angle regression. The Annals of statistics 

32(2):407–499.   

He D, Wu Y. 2014. A stochastic restricted principal components regression estimator in the linear model. 

The Scientific World Journal, vol. 2014, Article ID 231506, 6p. https://doi.org/10.1155/2014/231506 

Hoerl E, Kennard RW. 1970. Ridge regression: Biased estimation for nonorthogonal problems. 

Technometrics 12:55–67.  

Hubert MH, Wijekoon P. 2006. Improvement of the Liu estimator in linear regression model. Statistical 

Papers 47:471–479.   

Jibo W. 2014. On the Stochastic Restricted r-k Class Estimator and Stochastic Restricted r-d Class 

Estimator in Linear Regression Model. Journal of Applied Mathematics, vol. 2014, Article ID 

173836, 6 p. https://doi.org/10.1155/2014/173836 

Jibo W, Hu Y. 2014. On the stochastic restricted almost unbiased estimators in linear regression model. 

Communications in Statistics-Simulation and Computation 43(2):428–440.  

Kaçiranlar S, Sakallıoğlu S. 2001. Combining the Liu estimator and the principal component Regression 

Estimator. Communications in Statistics-Theory and Methods 30(12):2699–2705.   

Kayanan M, Wijekoon P. 2018. Stochastic Restricted Biased Estimators in Misspecified Regression 

Model with Incomplete Prior Information. Journal of Probability and Statistics, vol. 2018, Article 

ID 1452181, 8 p. https://doi.org/10.1155/2018/1452181.  

Kayanan M, Wijekoon P. 2019. Optimal estimators in misspecified linear regression model with an 

application to real-world data. Communications in Statistics: Case Studies, Data Analysis and 

Applications 4(3-4):151–163.   

Kayanan M, Wijekoon P. 2020a. Stochastic Restricted LASSO-Type Estimator in the Linear Regression 

Model. Journal of Probability and Statistics, vol. 2020, Article ID 7352097, 7 p. 

https://doi.org/10.1155/2020/7352097. 

Kayanan M, Wijekoon P. 2020b. Variable Selection via Biased Estimators in the Linear Regression 

Model. Open Journal of Statistics 10(1):113–126.    

Li Y, Yang H. 2010. A new stochastic mixed ridge estimator in linear regression model. Statistical Papers 

51:315–323.   

Liu K. 1993. A new class of biased estimate in linear regression. Communication in Statistics - Theory 

and Methods 22(2):393–402.   

Massy WF. 1965. Principal components regression in exploratory statistical research. Journal of the 

American Statistical Association 60:234–266. 

McDonald GC, Galarneau DI. 1975. A Monte Carlo Evaluation of Some Ridge-Type Estimators. Journal 

of the American Statistical Association 70:407–416.   

Nagar AL, Kakwani NC. 1964. The Bias and Moment Matrix of a Mixed Regression Estimator. 

Econometrica 32:174–182.   

Singh B, Chaubey YP, Dwivedi TD. 1986. An Almost Unbiased Ridge Estimator. The Indian Journal of 

Statistics 48:342–346. 

Stamey TA, Kabalin JN, Mcneal JE, Johnstone IM, Freiha F, Redwine EA, Yang N. 1989. Prostate 

specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. radical 

prostatectomy treated patients. Journal of Urology 141:1076–1083.  

Theil H, Goldberger AS. 1961. On Pure and Mixed Statistical Estimation in Economics. International 

Economic Review 2:65–78.  

Tibshirani R. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical 

Society: Series B (Methodological) 58:267–288.  

Zou H, Hastie T. 2005. Regularization and variable selection via the elastic net. Journal of the Royal 

Statistical Society: Series B 67:301–320. 

https://doi.org/10.1155/2014/231506


 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
25 

Appendix A: Figure 4 

 

(a) 

 

(b) 

 

(c) 

 

(d) 

 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
26 

 

(e) 

 

(f) 

 

(g) 

 

(h) 

 



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
27 

 

(i) 

Fig 4. Coefficient paths of the (a) LARS-MRE, (b) LARS-SRRE, (c) LARS- SRAURE, (d) 

LARS-SRLE, (e) LARS-SRAULE, (f) LARS-SRPCRE, (g) LARS-SRrk (h) LARS-SRrd 

and (i) LARS-SROE versus 𝒕 = ∑ |𝜷𝒋|
𝒑
𝒋=𝟏   for the Prostate Cancer Data. 

 

 

  



 M. Kayanan and P. Wijekoon                                                         Generalized Stochastic Restricted LARS Algorithm 

Ruhuna Journal of Science 

Vol 13 (1): 14-28, June 2022 
28 

Appendix B: Selection of prior information 

 

According to Nagar and Kakwani (1964), we can define the prior information as follows:  

Let 𝜷1 be a vector of some selected 𝑞 elements of 𝜷 and 𝜷2 is the rest of elements. Assume 
that 𝒃 is the known unbiased estimates of 𝜷1. By using the “two sigma rule”, now we can write 
the range of 𝜷1 as 𝒃 ± 2𝑆𝐸(𝒃). Based on that we can set the expressions of equation (2.2) as 

�̂� = 𝒃,   �̂� = (

1 0 ⋯ 0 ⋯ 0
0 1 ⋯ 0 ⋯ 0
⋮
0

⋮
0

⋱
⋯

⋮
1

⋯
⋯

0
0

)

𝑞×𝑝

, 𝜷 = (
𝜷1
𝜷2
)  and  

 𝜎2�̂� = (

𝑆𝐸(𝑏1) 0      ⋯      0

0
⋮
0

𝑆𝐸(𝑏2)
⋮
0

⋯
⋱
⋯

0
⋮

𝑆𝐸(𝑏𝑞)

)

𝑞×𝑞

. 

 

Appendix C: K-fold cross-validation to estimate shrinkage parameters, (k, d)  

 

Step 1: Split the data set into 𝐾 groups. 
Step 2: For each unique group 𝑖 =  1,2, . . . ,𝐾:  

• Take one group as a test data set 

• Take the remaining 𝐾 − 1 groups as a training data set 

• Estimate the respective estimator �̂�(𝑘,𝑑)  with shrinkage parameters (𝑘,𝑑) using 
the training data set. Use intital values of 𝑘,𝑑 as 0.01, and then compute the RMSE 
cross-validation errors separately as follows: 

𝐶𝑉_𝑅𝑀𝑆𝐸𝑖(�̂�(𝑘,𝑑))   = √
𝐾

𝑛
 ∑ (𝑦𝑖

∗ − ∑ 𝑥𝑖𝑗
∗ �̂�𝑗(𝑘,𝑑)

𝑝
𝑗=1

)
2

𝑛/𝐾
𝑖=1

, 

 

where 𝑦𝑖
∗  and 𝑥𝑖𝑗

∗   are the variables belongs to the test data set. 

Step 3: Then, find the overall cross-validation errors as follows: 

𝐶𝑉_𝑅𝑀𝑆𝐸 (�̂�(𝑘,𝑑)) = 
1

𝐾
∑𝐶𝑉_𝑅𝑀𝑆𝐸𝑖 (�̂�(𝑘,𝑑))

𝐾

𝑖=1

    

Step 4: Continue this procedure by increasing the values of 𝑘 or 𝑑 by a small increment, and 

choose the value of 𝑘 or 𝑑 that makes 𝐶𝑉_𝑅𝑀𝑆𝐸 (�̂�(𝑘,𝑑))  smallest.