IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 Bayesian Analyses of Ridge Regression Prooblems H. M. Gorgees Department of Mathematics, College of Education Ibn-Al-Haitham,University of Baghdad Abstract: A Bayesian formulation of the ridge regression problem is considerd, which derives from a direct specification of prior informations about parameters of general linear regression model when data suffer from a high degree of multicollinearity .A new approach for deriving the conventional estimator for the ridge parameter proposed by Hoerl and Kennard (1970) as well as Bayesian estimator are presented. A numerical example is studied in order to compare the performance of these estimators. Introduction: The problem of multicollinearity exists when there exists a linear relationship or an approximate linear relationship among two or more explanatory variables. Multicollinearity can be thought of as a situation where two or more explanatory variables in the data set move together. As a consequence it is impossible to use this data set to decide which of the explanatory variables is producing the observed change in the response variable. No treatment of the data or transformation of the model will cure this deficiency. Consequently,the best way to deal with multicollinearity may be to find a different data set, or additional data to break the association between the related variables. However some multicollinearity is nearly always present, but the important point is whether the multicollinearity is serious enough to cause appreciable damage to the regression. Indicators of multicollinearity include a low determinant of the information matrix, a very high correlation among two or more explanatory variables , very high correlations among two or more estimated coefficients, and significant regression of one explanatory variable on one or more explanatory variables. Key words: linear regressioin model , multicollinearity , ridge regression , generalized shrinkage estimators , Bayesian estimator , singular value decomposition. IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 This paper deals with multicollinearity in the classical linear regression model y=xβ+u …..(1) Where y is an ( of observations on the response variable, x = ( is an (n matrix and of full column rank, β is a ( p parameter vector (vector of unknown regression coefficients) and u is an ( n vector of random disturbances, E(u)=0 and var(u)= I, and both β and are unknown. The least squares estimator of β is: (see[1]) = x'y …..(2) Where denote the least squares estimator of β. The two key properties of are that it is unbiased,E( )=β, and that it has minimum variance among all linear unbiased estimators.The mean square error of is: MSE( )= ……(3) (see[2]) Where "s are the eigenvalues of x'x and ….. >0 .. If the smallest eigenvalue of x'x is very much smaller than 1, then a seriously ill_conditioned (or multicollinearity ) problem arises. Thus,for ill_conditioned data, the least squares solution yields coefficients whose absolute values are too large and whose signs may actually reverse with negligible changes in the data. That is in the case of multicollinearity the least squares estimator can be poor in terms of various mean squared error criterion. Consequently, a great deal of work has been done to construct alternatives to the least squares estimator when multicollinearity is present. In the seventies Hoerl and Kennard introduced a class of biased estimators for parameters in general linear regression model labeled ridge estimators as a rival to the least squares estimator when sample data are affected by a high degree of multicollinearity . The outhers show that in any given problem there is at least one member of this class which has total mean square error smaller than the total variance of the corresponding least squares estimator. The ridge estimator depends crucially upon an exogeneous parameter, say k. For any k the corresponding ridge estimator denoted by is defined to be: = x'y…….(4) ( see[1]) IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 We argue that it is typically true that there is available prior informations about the parameters, and this may be exploited to find improved estimators. In this paper attention is focuced upon Bayesian formulation for generalized shrinkage estimators and ordinary ridge regression estimator,moreover a Bayes estimator as well as a conventional estimator for the ridge parameter is derived. Generalized Shrinkage Estimators: Given an (n ) matrix of regressors x and an (n ) vector of the corresponding response y. Assume that sample means have been removed from the data (so that 1'x=0'and 1'y=0, where 1 is an n_vector of ones.) and write the standard linear regression model as E(y|x)=xβ and var(y|x)= (I-11'/n) where β is a vector of unknown regression coefficients and is the unknown error variance. The singular value decomposition of x will be denoted by: (see[3]). X=H G' …….(5) Where H is an ( semi orthogonal matrix satisfy H'H= , is a (p diagonal matrix of orderd singular values of x, ….. > 0, G is a (p orthogonal matrix whose columns represent the eigenvectors of the information matrix x'x. Assume that >0 so that β is estimable, then as in" Obenchain (1978)" (see [4]) we get =GC where C= H'y contains the uncorrelated components of where E(C)=E(G' ) = G'β =γ say, and: Var(C)=var(G' )=G'var( )G= G' G= . Notice that the elements of C are uncorrelated since their variance matrix is diagonal. The vector of generalized shrinkage estimator (or generalized ridge regression estimator) will be denoted here by and will be of the general form : (see[4]) =G∆C= …….. (6) Where is the j_th column of the matrix G, is the j_th diagonal element of the shrinkage factors matrix ∆, we will usually restrict the range of shrinkage factors to 0 , j= 1,2, ….,p. is the j_th element of the uncorrelated components vector C. In the remaining of this section we discuss Bayesian methods for defining the form of shrinkage of sample estimates towards a subjective prior distribution.Lindely and Smith (1972) describe a Bayesian formalizim for hierarchical (multi_stage) analysis of linear models using conjugate IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 multivariate_normal prior distribution.(see [5]) This formalism expresses unknown parameters at each stage of an analysis in terms of a linear model at the previous lower stage . But, although dispersion matrices at each stage can be arbitrary , they must be known. And, at the final stage, both the mean vector and the dispersion matrix must be known.The fundamental lemma of Lindely and Smith (1972) states that: Lemma:If the sampeling distribution of the response,y, is y| ~ N( , ) where is ( ×1) parameter vector and the prior distribution is ~ N( , ) where is ( ×1) parameter vector, then the marginal (unconditional) distribution of y is: y ~ N( , + ') ……(7) and the posterior (conditional) distribution of given y is: |y ~ N(Bb , B) ………(8) Where : = ' + and b= ' y+ To apply this lemma and demonstrate that a simple 2_stage Bayesian formalism produces generalized shrinkage estimators, we first make the identification: = xβ , = I and = x'x + . Next, we set the prior mean value for β to zero by taking =0 and assume that (and ) will be simultaneously diagonalizable with x'x by restricting attention to prior variance_covariance matrices of the general form = .G G', where K is a diagonal p×p matrix and G is defined as in (5) . Now the Bayes estimate is the mean Bb of the posterior distribution of β given y and this mean vector is of the general form: E(β|y)=Bb= ( ' Y+ ) = = = G H'y =G H'y IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 =G D H'y = G∆C = …….(9) Where ∆ = D is the diagonal matrix of generalized shrinkage factors and C is the vector of uncorrelated components of the least squares estimator. The Ordinary Ridge Regression Estimator: In the previous section we have demonstrated that all generalized shrinkage estimators are 2_stage Bayes estimators. This include of course the case of ordinary ridge regression estimator p roposed by Hoerl and Kennard (1970). To demonstrate this fact let us assume that an orthogonal matrix P is given such that P' P = D = diag( ,….., ), ≥…….≥ >0. Morever, suppose that Z = P' = ( ,….., )' and W=P'β = ( ,…., )' then: Z ~ (W, D). Let us assume that W has a prior distribution given by W ~ (0, λI) for some positive constant λ, thus, according to the lemma stated in section 2, the posterior distribution of W given Z is: W|Z ~ [ λ Z, λI- ] …….(10) then the Bayes estimator of W is: = Z for Z=P' . Consequently, the Bayes estimator of β is: P = P P' = = = x'y for k = = x'y = ……(11). It is obvious that the estimator in (11) coincide with the ordinary ridge regression estimator given in (4). Estimating The Ridge Parameter_Bayesian Approach: There are many different methods for selecting the value of the ridge parameter k. The method we try here is based upon the lemma stated in section (2). Accordingly,the marginal distribution of Z is: Z ~ (0, D+ λI) IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 E(Z' Z) = tr var(Z) = tr ( D+ λ I) = tr +λ tr …..(12) where "tr" denotees the trace of the matrix.To find an unbiased estimator for λ, the following result is necessary. (see[6]) E( = ……..(13) Where is an estimator of . The result in (13) can be easily proved by setting (n-p- 1) / =r then r ~ E( ) = E( ) = (see [7] page 176) From (12) and (13) an unbiased estimator for λ can be obtained as: = [ ( ) - p]/tr or equivalently: = = [ ( ) – P]/tr(x'x) ……….(14) Estimating the ridge parameter_New Approach: Substituting from Δ in formula (6) by D it can easily be shown that the ordinary ridge regression estimator is a member of generalized shrinkage estimators class, in such a case the shrinkage factors of ordinary ridge regression estimator will have the form : = , 1 ………(15) In (1970) Hoerl and Kennard proposed an estimator for the ridge parameter given as: = ……..(16) [ see[ 8 ]) In this section, a new approach is used to derive given in (16) , this approach is represented by minimizing MSE( ) as follows: MSE( ) = MSE(GΔC) = GMSE(ΔC)G' Where MSE(ΔC) = +(I-Δ)γγ'(I-Δ) Is the mean squared error matrix of ΔC with ith diagonal element given as: { see[3]} IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 MSE( ) = / + ……….(17) Differintiating MSE( ) with respect to we obtain: =2 / -2(1- ) ………. (18) While the second partial derivative is nonnegative constant it follows that equating the derivative in (18) to zero will yield a minimum value for MSE( ) this optimal amount of shrinkage for the i_th uncorrelated component is: = = …….. (19) We derive the formula in (19) as follows: 2 / -2(1- ) =0 then / = - hence = - which implies that + = then solving for we obtain the required result. Now we suggest comparing the shrinkage factor of ridge regression estimator given in equation (15) with given in equation (19), hence we conclude that the value of the ridge parameter k must be equal to / . Since each of and is unknown, we can use their estimated values, thus : = = ……….(20) Where is the residual mean square in the analysis of variance table obtained from the standard least squares fit. Numerical Example: In this section a data set suffer from a high degree of multicollinearity is used. The data are from Aljibori (2004) (see [11] for details). It was designed to measure the effect of five explanatory variables , ,…., on the response variable y. Where the explanatory variables represent the number of , managerials , technicians, skilled workers, unskilled workers and service workers respectively, while the response variable y represents the productivity of the industrial sector in Iraq measured by the surplus value method for the period 21 years from 1970 to 1990.Our purpose is only to compare the performance of Bayesian and conventional estimators for the ridge parameter. The original data are presented in table (1). IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 For this data we found that the estimated value of the ridge parameter was = 0.125260 and = 0.494107 obtained by applying the formula in (14) and in (20) respectively. In order to make the ridge regression analysis we used the numeric calculation statistical system (NCSS) and the results were given in table (2) through table (6). Pearson correlations were given for all variables in table (2). These correlation coefficients show which explanatory variables are highly correlated with response variable and with each other. Explanatory variables that are highly correlated with one another may cause multicollinearity problem. Table (3) gives an eigenvalue analysis of the explanatory variables after they have been centerd and scaled. Notice that incremental percent is the percent this eigenvalue is of the total. Percents near zero indicate a multicollinearity problem. The condition number is the largest eigenvalue divided by each corresponding eigenvalue. Condition numbers more than 100 indicates multicollinearity problem. (See[9]). Conclusions: 1_A new approach for estimating the ridge parameter was introduced by using the singular value decomposition technique. 2_In their development of ridge regression, Hoerl and Kennard focus attention on the eigenvalues of the information matrix x'x. A seriously non orthogonal problem is characterized by the fact that the smallest eigenvalue is very much smaller than unity . In our problem the smallest eigenvalue is 0.0093 this indicates that our data set suffer from a high degree of multicollinearity . 3_It should also be noted that the variance inflation factor (VIF) is an additional measure of multicollinearity . It is the reciprocal of (1- ) where is the square value of the multiple correlation coefficient between the explanatory variable and other explanatory variables. A VIF of 10 or more indicates a multicollinearity problem. (See[10]). In our problem the largest VIF value is 65.16 which is an additional indicator that our data set suffer from a high degree of multicollinearity . 4_ Since one of the objects of ridge regression is to reduce the standard error of the regression coefficients, it is of interest to see how much reduction has taken place. For our problem it should be noted from table (5) and table (6) that the standard errors for the ridge regression coefficients are less than the corresponding standard errors for least squares coefficients. Also we note that the standard errors for the ridge regression coefficients obtained by using the conventional mehod for estimating the ridge parameter are less than the corresponding standard errors for the ridge regression coefficients obtained by using the Bayesian method. From this comparison we conclude that the conventional mehod is performed better than the Bayesian method for this data set. IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 References: 1. Draper, N.R. and Smith, H. (1981),"Applied regression analysis" second edition, John Wiley and Sons, New York. 2.Goldstein,M. and Smith,A.F.M.(1974),"Ridge type estimator for regression analysis."Jornal Royal Statistical Society. B36,284_291. 3.Rao,C.R.(1973),"Linear Statistical Inference and Its Applications."second edition,John Wiley and Sons. New York. 4.Obenchain,R.L. (1978),"Good and optimal ridge estimators." Ann. Statist.6,1111_1121. 5.Lindley,D.V. and Smith,A.F.M.(1972),"Bayes estimators for the linear model."Jornal Royal Statistical Society. B34, 1_41. .Srivastava,M.I. (2002),"Methods Of Multivariate Statistics."Wiley, New York6 7.Hogg,R.V. and Craig,A.T.(1978),"Introduction to mathematical statistics." Fourth edition,Macmillan pub.Co.New York. 8.Hoerl,A.E. and Kennard,R.W.(1970),"Ridge regression: Biased estimation for non , 55_67.12orthogonal problems."Technometrics. 9.Chatterjee,S. and Price,P.(1977),"Regression analysis by example."John Wiley and Sons, New York. 10.Marquardit,D.W.(1970),"Generalized inverse,Ridge regression, and nonlinear , 591_612.12estimation."Technometrics. تخطیط الموارد البشریة ودورھا في تنمیة القطاع الصناعي في العراق ) : 2004.( خالد ابراھیم سلمان، الجبوري 11. رسالة ماجستیر مقدمة الى مجلس المعھد العالي للدراسات السیاسیة والدولیة في الجامعة . 1990_1970للفترة .المستنصریة IHJPAS IBN AL- HAITHAM J. FOR PURE & APPL. SCI. VO L.23 (3) 2010 Table(1)Effects of five explanatory variables x1,x2,….,x5 the response variable y on Table(2) Correlation matrix section Y X5 X4 X3 X2 X1 0.749380 0.821688 0.380475 - 0.780634 0.967806 1.000000 X1 0.790267 0.912351 -0.541955 0.688120 1.000000 0.967806 X2 0.631133 0.513239 0.028945 - 1.000000 0.688120 0.780634 X3 0.295671 - 0.645230 - 1.000000 0.028945 - 0.541955 - 0.380475 - X4 0.806230 1.000000 0.645230 - 0.513239 0.912351 0.821688 X5 1.000000 0.806230 0.295671 - 0.631133 0.790267 0.749380 y Table(3) Eigen values of correlations X5 X4 X3 X2 X1 y 14216 50096 24524 473 1516 1271 16467 57133 26012 1923 1514 1440 18697 66795 30360 2490 2037 1355 18716 66263 29385 2734 2019 1509 20696 61718 36619 3322 2210 1254 23474 61028 37393 3563 2234 1220 25283 69344 40034 4205 2525 1546 26172 66181 10310 4293 2381 1916 28108 62039 49163 5278 2944 2381 30707 69339 57299 8529 3528 2585 32561 69671 60965 9661 4308 2810 33609 69047 56600 9840 4444 1440 32789 65441 55789 10830 4588 2493 52340 44175 44125 12968 4075 3285 47633 52476 46492 12875 4563 3062 63398 48322 46599 12163 4031 3403 62205 49672 49070 12712 4479 2875 47530 47330 49035 12610 4343 2861 46260 46160 48013 12615 4299 2596 4510 45123 40860 12630 4345 1710 44915 44925 49095 12955 4865 1777 Condition number Cumulative percent Incremental percent eigenvalue No. 1.00 72.47 72.47 3.623598 1 3.45 93.51 21.04 1.051811 2 7.151 97.73 4.22 0.211248 3 34.83 99.81 2.08 0.104043 4 389.65 100.00 0.19 0.009300 5 IHJPAS IBN AL- HAITHAM J. FO R PURE & APPL. SC I. VO L.23 (3) 2010 Table(4)Eigenvectors of correlations X5 X4 X3 X2 X1 eigenvalue No. 0.487824 - 0.304255 0.284390 - 0.518719 - 0.502625 - 3.623598 1 0.210802 0.751940 - 0.592083 - 0.007497 - -0.199033 1.051811 2 0.425416 0.569859 0.539120 - 0.222736 0.190854 0.211248 3 0.710381 - 0.109447 - 0.303237 - 0.292886 0.552851 0.104043 4 0.179037 0.074329 - 0.035429 - 0.771674 - 0.604718 0.009300 5 Table(5)Ridge vs. Least squares comparison section for k = 0.125260 LS standad error Ridge standard error LS VIF Ridge VIF Standarded LS coefficients Standardized Ridge coefficients Explanatory variables 0.4965 0.0884 42.54 0.955 0.9859 - 0.0126 X1 0.1562 0.0184 65.16 0.637 1.1247 0.2329 X2 0.0127 0.0093 3.326 1.248 0.2634 0.1695 X3 0.0154 0.0114 2.808 1.085 0.4680 0.1653 X4 0.0181 0.0085 9.261 1.447 0.7006 0.5359 X5 Table(6)Ridge vs. Least squares comparison section for k = 0.494107 LS standard error Ridge standard error LS VIF Ridge VIF Standardized LS Coefficients Standardized Ridge Coefficients Explanatory variables 0.4965 0.0447 42.54 0.1892 -0.9859 0.1319 X1 0.1562 0.0092 65.16 0.1254 .12471 0.1970 X2 0.0127 0.0059 3.326 0.3861 0.2634 0.1563 X3 0.0154 0.0079 2.808 0.4101 0.4680 0.0452 X4 0.0181 0.0044 9.261 0.2951 0.7006 0.3126 X5 IHJPAS 2010 )3( 23مجلة ابن الھیثم للعلوم الصرفة والتطبیقیة المجلد توظیف اسلوب بیز في تحلیل انحدار الحرف حازم منصور كوركیس جامعة بغداد،ابن الھیثم -كلیة التربیة،قسم الریاضیات خالصھال فر معلومات مسبقة عن احرف على فرض تواسلوب بیز في تحلیل انحدار الحرف وتقدیر معلمة ال وظففي ھذا البحث كما تم ، وان االنموذج یعاني من مشكلة التعدد الخطي غیر التام بدرجة كبیرة ،معلمات انموذج االنحدار الخطي العام ومن 1970في عام Hoerl and Kennardاستخدام اسلوب جدید في ایجاد مقدر معلمة الحرف الذي اقترحھ كل من .عددي اجریت مقارنة ألفضلیة اداء ھذه المقدرات خالل دراسة مثال IHJPAS