This paper proposes a new biased estimator for estimating the regression parameters for the multiple linear regression models when the regressors are correlated. Theoretical comparisons and simulations results show that, the proposed estimator performs better than other existing estimators under some conditions in the smaller mean squares error sense. A real-life dataset is analyzed to illustrate the findings of the paper.

Biased estimator, Liu estimator, Monte Carlo simulation, Multicollinearity, MSE, Ridge regression estimator

The ordinary least squares (OLS) estimator is the best linear unbiased estimator and has been used to estimate the parameters of the linear regression model since its inception. One of the important assumptions for the linear regression model is that the regressors (independent variables) are independent. However, in practice the regressors may or may not be independent, which causes the problem of multicollinearity. In the presence of the multicollinearity, the OLS estimator is inefficient and gives wrong sign of the parameters in the multiple linear regression models [1]. To handle these problems, many authors have given different types of estimators, to mention a few, [1-7], and recently [8], for one parameter biased estimator. However, to refer for two parameters mode, the following authors are notable, [9-15], among others.

The main objective of this paper is to propose a new two parameter biased estimator for the regression coefficients and then to compare the performance of the new estimator with the OLS, the ordinary ridge regression (ORR) of Hoerl and Kennard [1], the Liu of Liu [6] and the Kibria-Lukman (KL) of Kibria and Lukman [8] estimators. The rest of the paper is organized as follows: Some estimators and their statistical properties are given in Section 2. In section 3, the theoretical comparisons among the proposed estimator and existing estimators and the biasing parameters k and d are given. A Monte Carlo simulation study is performed in section 4. A real-life data are analyzed in section 5. Finally, Some Conclusions are given in section 6.

Consider the following linear regression model:

y=Xβ+ε, (2.1)

where y is an n×1 vector of the response variable, X is a known n×p full rank matrix of the explanatory variables, β is an p×1 vector of unknown regression coefficients, ε is a n × 1 vector of disturbance assumed to be distributed with mean vector 0 and variance covariance matrix σ2I, and I is an identity matrix of order n × n. To define various estimators, canonical form of the model (2.1) is given by:

y=Zα+ε, ε~N(0, σ2In) (2.2)

where, Z=X C, α=C′ β, and C is an orthogonal matrix such that Z′ Z=C′X′ X C=U=diag (u1, u2, …, up). Then, the OLS estimator of α is given as:

ˆ α =U−1Z′ y, (2.3)

and then the mean squared error matrix (MSEM) of ˆ α is given by MSEM( ˆ

α )=σ2U−1. (2.4) The ORR of α [1] is

ˆ α k= N U ˆ α , (2.5)

Where, N=[U+kI]^{-1} and the MSEM of
ˆ

α k would be

MSEM( ˆ α k)=σ2 NU N′+ (NU−Ip) α α′ (NU−Ip)′ (2.6)

Hoerl, et al. [16] deﬁned the biasing parameter k for ˆ α

k as follows:

ˆ k HM= p ˆ σ 2 p ∑ i=1 ˆ α 2 i . (2.7)

The Liu estimator of α [6] is

ˆ α d= F ˆ α , (2.8)

Where, F =[U+Ip]−1[U+d Ip] and the biasing parameter d of

ˆ α d is given by ˆ d opt=1− ˆ σ 2[ p ∑ i=1 1/(ui(ui+1)) p ∑ i=1 ( ˆ α 2 i /(ui+1)2) ] (2.9) MSEM( ˆ

α d)=σ2 F U−1F′+ (1−d)2(U+Ip)−1αα′(U+Ip)−1. (2.10) If ˆ d

opt is negative, Ozkale and Kaciranlar [9] adopt the alternative biasing parameter below:

ˆ d alt=min[ ˆ α 2 i ( ˆ σ 2/ui)+ ˆ α 2 i ] p i=1 . (2.11)

The recently proposed KL estimator by Kibria and Lukman [8] is defined as,

ˆ α KL= NM ˆ α , (2.12)

Where, M=(U−k Ιp) and the biasing parameter k of the KL estimator is defined by

ˆ k min(KL)=min [ ˆ σ 2 2 ˆ α 2 i +( ˆ σ 2/ui) ] (2.13) MSEM( ˆ α

KL)=σ2 NM U−1M′ N′ +[ NM −Ip] α α′ [ NM −Ip]′ (2.14)

The new biased (NB) estimator of α is obtained by minimizing (y−Zα)′(y−Zα) subject to

( √ k α+ √ k (1+d) ˆ α )′( √ k α+ √ k (1+d) ˆ

α )=c where c is a constant,

(y−Zα)′ (y−Zα)+ [( √ k α+ √ k (1+d) ˆ α )′( √ k α+ √ k (1+d) ˆ α )−c] . (2.15)

Here, k and d are the Lagrangian multipliers.

The solution to (2.15) gives the new estimator as follows:

ˆ α NB=(U+k Ιp)−1(U−k (1+d) Ιp) ˆ α =NB ˆ α (2.16)

where B=(U−k (1+d) Ιp), k≥0 and 0<ld<l1.

Moreover, the proposed NB estimator is also obtained by augmenting −

√ k (1+d) ˆ

α = √ k

α+ε′ to equation (2.2) and then using the OLS estimate.

The proposed NB estimator is a general estimator which includes the OLS and the KL estimator:

1 If k=0 in ˆ

α NB becomes ˆ

α .

2 If d=0 in ˆ

α NB becomes ˆ

α KL.

The MSEM of the proposed NB estimator of α is given by

MSEM( ˆ α NB)=σ2NB U−1B′N′+[ NB−Ip] α α′ [ NB −Ip]′ . (2.17)

The lemmas below will be used for theoretical comparisons among estimators in the next section.

Lemma 2.1: [17] Let E be an n×n positive definite matrix, that is E>0 and α be some vector; then, E−α α′>0 if and only if α′ E−1α<1.

Lemma 2.2: [18] Let αi=Ai y, i=1, 2 be two linear estimators of α. Suppose that D=Cov ( ˆ

α 1)−Cov ( ˆ

α 2)>0, where Cov ( ˆ

α i) i=1, 2 be the covariance matrix of

ˆα i and bi=Bias( ˆ

α i)=(AiX−I)α, i = 1,2. Consequently,

Δ( ˆ

α 1− ˆ

α 2)=MSEM( ˆ

α 1)−MSEM( ˆ

α 2)=σ2D+b1b′1−b2b′2>0 (2.18)

if and only if b′2[σ2D+b′1b1]−1b2>1 where MSEM( ˆ

α i)=Cov ( ˆ

α i)+bib′i.

Comparison between ˆ α and ˆ

α NB Theorem 3.1: MSEM ( ˆ

α )−MSEM ( ˆ

α

NB)>0 if and only if

α′[ NB −Ip]′ [σ2 (U−1−NB U−1B′N′)] [ NB−Ip] α <1 (3.1)

Proof:

D( ˆ α )−D( ˆ α NB)=σ2 (U−1−NB U−1B′N′) =σ2 diag { 1 ui

− (ui−k (1+d))2 ui(ui+k)2 } p i=1 (3.2)

Where U−1−NB U−1B′N′ will be positive definite (pd) if and only if (ui+k)2− (ui−k (1+d))2>0 or (ui+k)− (ui−k (1+d))>0. Clearly, for k>0 and 0<d<1, (ui+k)− (ui−k (1+d))=k (2+d) >0. By Lemma 2.2. The proof is completed.

Comparison between ˆ

α k and ˆ

α NB Theorem 3.2: MSEM ( ˆ

α k)−MSEM ( ˆ

α

NB)>0 if and only if

α′[ NB −Ip]′ [V1 + (NU−Ip) α α′ (NU−Ip)′] [ NB −Ip] α <1 (3.3)

Where, V1=σ2 (NUN′−NB U−1B′N′)

Proof:V1=σ2 (NUN′−NB U−1B′N′) =σ2 diag { ui (ui+k)2 − (ui−k (1+d))2 ui(ui+k)2 } p i=1 (3.4)

Where, NUN′−NB U−1B′N′ will be pd if and only if u 2 i

− (ui−k (1+d))2>0 or ui− (ui−k (1+d))>0. Clearly, for k>0 and 0<d<1, ui− (ui−k (1+d))= k (1+d)>0. By Lemma 2.2. The proof is completed.

Comparison between ˆ α d and ˆ

α NB Theorem 3.3: MSEM ( ˆ

α d)−MSEM ( ˆ

α

NB)>0 if and only if

α′[ NB −Ip]′ [V2 + (F−Ip) α α′ (F−Ip)′] [ NB −Ip] α <1 (3.5)

where, V2=σ2 (FU−1F−NB U−1B′N′)

Proof:

V2=σ2 (FU−1F−NB U−1B′N′) =σ2 diag { (ui+d)2 ui(ui+1)2 − (ui−k (1+d))2 ui(ui+k)2 } p i=1 (3.6)

Where, FU−1F−NB U−1B′N′ will be pd if and only if (ui+d)2(ui+k)2− (ui+1)2 (ui−k (1+d))2>0 or (ui+d)(ui+k)− (ui+1) (ui−k (1+d))>0. Clearly, for k>0 and 0<d<1, (ui+d)(ui+k)− (ui+1) (ui−k (1+d))=ui (2k+kd+d−1)+k (2d+1)>0. By Lemma 2.2. The proof is completed.

Comparison between

ˆ α KL and ˆ

α NB Theorem 3.4: MSEM ( ˆ

α KL)−MSEM ( ˆ

α

NB)>0 if and only if

α′[ NB −Ip]′ [V3 + (NM−Ip) α α′ (NM−Ip)′] [ NB −Ip] α <1 (3.7)

Where, V3=σ2 (NMU−1M′ N′−NB U−1B′N′)

Proof:

V2=σ2 (NMU−1M′ N′−NB U−1B′N′) =σ2 diag { (ui−k)2 ui(ui+k)2 − (ui−k (1+d))2 ui(ui+k)2 } p i=1 (3.8)

Where, NMU−1M′N′−NB U−1B′N′ will be pd if and only if (ui−k)2− (ui−k (1+d))2>0 or (ui−k)− (ui−k (1+d))>0. Clearly, for k>0 and 0<d<1, (ui−k)− (ui−k (1+d))=k d>0. By Lemma 2.2 The proof is completed.

Different biasing parameters estimators of k and d are proposed in different studies, for example, Hoerl and Kennard [1], Liu [6], [16,19-23], among others.

So the optimal values of k and d for the proposed NB estimator is going to be found. At first, by minimizing the equation m, we obtain the optimal value of k when d is fixed as

MSEM( ˆ α NB)=E (( ˆ α NB−α)′ ( ˆ α NB−α)), m = tr (MSEM( ˆ α NB)), m = σ2 p ∑ i=1 (ui−k (1+d))2 ui (ui+k)2 +k2(2+d)2 p ∑ i=1 α 2 i (ui+k)2 (3.9)

Differentiating m with respect to k and setting (∂m/∂k)=0, we get

k= ui σ2 (d+1) (ui α 2 i +σ2)+ui α 2 i . (3.10)

Then, the estimated optimal value of k is given as follows:

ˆ k = ui ˆ σ 2 (d+1) (ui ˆ α 2 i + ˆ σ 2)+ui ˆ α 2 i , (3.11)

and,

ˆ k min(NB)=min { ui ˆ σ 2 (d+1) (ui ˆ α 2 i + ˆ σ 2)+ui ˆ α 2 i } p i=1 . (3.12)

Also, the optimal value of d will be found by differentiating m with respect to d when k is fixed and setting (∂m/∂d)=0, we get

d= ui σ2−(2 uiα 2 i +σ2) k k (ui α 2 i +σ2) . (3.13)

Then, the estimated optimal d with the unbiased estimators is given by

ˆ d = ui ˆ σ 2−(2 ui ˆ α 2 i + ˆ σ 2) ˆ k ˆ k (ui ˆ α 2 i + ˆ σ 2) , (3.14)

And

ˆ d min(NB)=min{ ui ˆ σ 2−(2 ui ˆ α 2 i + ˆ σ 2) ˆ k min(NB) ˆ k min(NB) (ui ˆ α 2 i + ˆ σ 2) } p i=1 . (3.15)

The parameters k and d estimators selection in ˆ

α

NB are found iteratively as follows:

1. Find an initial estimate of d using ˆ d =min ( ˆ σ 2 ˆ α 2 i ).

2. Determine ˆ k min(NB) from (3.12) using ˆ d in step 1. 3. Estimate ˆ

d min(NB) in (3.15) by using ˆ

k min(NB) in step 2.

4. If ˆ d min(NB) > 1 or ˆ d min(NB) < 0, use ˆ d min(NB)= ˆ d .

A Monte Carlo simulation study is performed to show the performance of the NB estimator over some existing estimators. It contains two parts: (i) Simulation technique and (ii) Results discussion.

Using the equation below, we generate the explanatory variables (see, Gibbons [24] and Kibria [19]):

xij=(1−ρ2)1/2zij+ρ zi, p+1, i=1, 2, ..., n, j=1, 2, ..., p (4.1)

where zij are independent standard normal pseudo-random numbers and ρ is the correlation between any two explanatory variables and ρ here has two values 0.9 and 0.99. The n observations for the response variable y are gotten by the following equation:

yi=β1xi1+β2xi2+β3xi3+ei, i=1, 2, …, n (4.2)

where ei are i.i.d N(0, σ2). The values of β are obtained as β′ β=1 [25]. Also, we choose the values of the biasing parameters of the estimators as k=d=0.1, 0.2, …,0.9 by following the work of Wichern and Churchill [26] and Kan, et al. [27] is that when k lies between 0 and 1, the ORR estimator performs better. The Monte Carlo simulation study replication is 1000 times for n = 50 and 100 and σ2= 1, 25, and 100. For each replicate, we calculate the mean square error (MSE) of the estimators using the next equation:

MSE(α*)= 1 1000 1000 ∑ j=1 (α * ij −αi)′(α * ij −αi) (4.3)

where α

*

ij

is the estimator and αi is the true parameter. The estimated MSEs of the estimators are shown in Table 1, Table 2, Table 3 and Table 4 for (ρ = 0.90 and n = 50), (ρ = 0.99 and n = 50), (ρ = 0.90 and n = 100), and (ρ = 0.99 and n = 100), respectively.

From Table 1, Table 2, Table 3 and Table 4, we observed that, when the factors σ and ρ are going to increase, the estimated MSE values are also going to increase, while n is going to increase, the estimated MSE values are going to decrease. Also, the OLS estimator is performing the worst for all cases in presence of the multicollinearity. Moreover, the simulation results show that, the proposed NB estimator is performing better than the other estimators for most of cases. The Liu estimator gives better results in the MSE values when k=d=0.1, 0.2 i.e. when the biasing parameters are near to zero. For ρ = 0.9: The condition number (CN) is approximately around 5 and the variance inflation factors (VIFs) are around 4 to 6 such that it is observed that a close agreement of the proposed NB (although better results) with the KL estimator for low values of k = d and a better performance as k = d increases for a fixed value of σ. This improvement increases with the increase of the value of σ and an even better improvements is observed if ρ = 0.99 where the CN and the VIFs become larger and are approximately around 15 and around 36 to 61, respectively. So, the proposed NB estimator works better for the strong correlated explanatory variables. So, the performance of the proposed NB estimator almost depends on the value of ρ, σ, the biasing parameters k and d, and the true parameter. Thus, simulation results are consistent with the theoretical results.

To illustrate the theoretical and simulation results of this paper, we consider a real life data in this section. The Portland cement data was originally adopted by Woods, et al. [28]. This data was also analyzed by many researchers, for examples, [29,30]; Lukman, et al. [14] and recently by Kibria and Lukman [8], among others. And this data is analyzed here to explain the performance of the proposed NB estimator and the other existing estimators. The regression model for this data is given by

yi=β0+β1X1+β2X2+β3X3+β4X4+εi. (5.1)

For knowing more details about this data, we refer Woods, et al. [28].

To show the existence of the multicollinearity, different measures are calculated as the VIFs which are 38.50, 254.42, 46.87 and 282.51, Eigenvalues of X'X are 44676.206, 5965.422, 809.952 and 105.419 and the CN of X'X approximately equals 20.58. The VIFs, the eigenvalues and the CN tell us that there is a severe multicollinearity in the data. Also, ˆ

σ

is equal to 2.446. The correlation coefficients matrix of the explanatory variables are presented in Table 5 such that there is a significant and strong relationship among the following explanatory variables: X1 and X3, X2 and X4. Then, the estimated parameters and the MSE values of the estimators are presented in Table 6. It appears from Table 6 that the proposed NB estimator is performing the best where it is giving an obvious improvement over all existing estimators and a little improvement over the KL estimator in which this is consistent with the simulation results because

ˆ

σ

here is small and not all the explanatory variables are significantly or strongly correlated at the same degree of correlation even though the data has high CN and VIFs.

In this paper, we proposed a new biased (NB) estimator for handling the multicollinearity problem in the multiple linear regression models. Some existing estimators are the special case of the proposed estimator. The proposed NB estimator is compared theoretically with the Ordinary least squares (OLS) estimator, the Ordinary ridge regression (ORR) estimator, the Liu estimator and the Kibria-Lukman (KL) estimator, and then the biasing parameters d and k of the NB estimator are derived. A Monte Carlo simulation study is performed for comparing the performance of the OLS, ORR, Liu, KL and the proposed NB estimators. The main finding of this simulation is that the proposed NB estimator performed better than the above mentioned estimators under some conditions. A real-life data is analyzed to support the findings of the paper.

Authors are grateful to referees for their valuable comments and suggestions, which certainly improved the quality and presentation of the paper. Author B M Golam Kibria would like to dedicate this paper to his Ph. D. supervisor, Late Professor M Safiul Haq for his guidance and constant inspiration during his stay at The University of Westen Ontario and finally to achieve this present position.