3.9 Tests and hypothese

The multiple regression model evolves several hypotheses that should be tested, such as:

The overall significance of the model.
The significance of every single coefficient.
The presence of some restrictions in some parameters, etc.

In this section, we will illustrate the tests associated with those hypotheses based on the OLS assumptions and some popular probability distributions such as Student, Fisher, Chi-square, etc. But before that, we should first introduce the famous criterion accompanying all the regression models, the coefficient of determination \(R^2\) that provides an approximate first judgment about the goodness of fit. However, be aware that this is not a statistic test since not derived from any probability theory.

3.9.1 The coefficient of determination \(R^2\)

Based on the equation of the sums ((3.24)), this criterion measures the percentage of the explained part from the total variation so that high values for \(R^2\) indicates a good fit. It can be expressed in terms of \(SSE=\widehat y^tN\widehat y\):

\[\begin{equation} R^2=\frac{\widehat y^tN\widehat y}{y^tNy}=\frac{\widehat y^t\widehat y-n\overline y^2}{y^ty-n\overline y^2} \tag{3.59} \end{equation}\]

Or in terms of \(SSR=e^te\):

\[\begin{equation} R^2=1-\frac{e^te}{y^ty-n\overline y^2} \tag{3.60} \end{equation}\]

Since the denominator of the the last expression is fixed (computed from the sample), the \(R^2\) value then depends only on \(e^te\).

The inconvenience of this criterion is that it never decreases with the addition of new regressors in the model, even though they are not relevant. Because, as noted in the section 3.8.2, the sum squared of errors \(e^te\) never decreases with the addition of new regressors. However, another version of this criterion exists that imposes a penalty for adding other regressors to the old version so that we end up with the so-called the adjusted R^2:

\[\begin{equation} \overline R^2=1-\frac{n-1}{n-k}(1-R^2) \tag{3.61} \end{equation}\]

The penalty is the factor \(\frac{n-1}{n-k}\leqslant1\) that reduces the value of \(R^2\) each time a new regressors are added.

It should be noted here that both \(R^2\) and \(\overline R^2\) are not considered as statistic tests because they are not related to any probability theory. The statistic tests will
be discussed in later chapters.

3.9.2 Hypothesis Formulation

In the previous chapter, we were only interested in the significance of the explanatory variable as it was the only one in the simple linear model. In contrast with the multiple linear models where other hypotheses appear. The most important ones are the following:

\(HO:\beta_i=0\): The significance of every single regressor separately.
\(HO:\beta_i=c\): The population parameter equals to the given value \(c\).
\(HO:\beta_i+\beta_j=c\): The sum of the two parameters equals to the given value \(c\). For instance, in the Cob Douglass production function, the sum of the capital elasticity and labor elasticity equals the constant returns to scale that is equal to 1.
\(HO:\beta_i=\beta_j \iff \beta_i-\beta_j=0\): The two parameters are equal meaning that the associated variables have the same effect on the dependent variable.
\(HO:\begin{pmatrix}\beta_2\\..\\\beta_k\end{pmatrix}=\begin{pmatrix}0\\..\\0\end{pmatrix}\): The overall significance of the model. Note that the constant term is omitted.
\(HO:\beta_{II}=0\): where \(\beta=\begin{pmatrix}\beta_I=\underset{(k_1\times 1)}{\begin{pmatrix}\beta_1\\..\\\beta_{k_1}\end{pmatrix}}\\\beta_{II}=\underset{k_2\times 1}{\begin{pmatrix}\beta_{k_1+1}\\..\\\beta_k\end{pmatrix}}\end{pmatrix}\).

The significance of some regressors as a whole defined by \(\beta_{II}\).

All those hypotheses can be grouped in a single equation in which the indices vary with every hypothesis as follows:

\[\begin{equation} \underset{(q\times k)}{R}\underset{(k\times 1)}{\beta}=\underset{(q\times 1)}{r} \tag{3.62} \end{equation}\]

the matrix \(R\) is called the restriction matrix.

Example 3.5 For simplification, suppose that we have the following DGP model:

\[\begin{equation*} y_t=\beta_1+\beta_2X_{2t}+\beta_3X_{3t}+\beta_4X_{4t}+\beta_5X_{5t}+\varepsilon_t \end{equation*}\]

Along with the following hypotheses:

\(HO:\beta_3=0\): The significance of the regressor \(X_3\).
\(HO:\beta_4=1\): The coefficient of the regressor \(X_4\) is equal to 1.
\(HO:\beta_2+\beta_5=3\): The sum of the two regressors \(X_2\) and \(X_5\) is equal to 3.
\(HO:\beta_2=\beta_4\iff \beta_2-\beta_4=0\): The two regressors \(X_2\) and \(X_4\) have the same effect on the dependent variable.
\(HO:\begin{pmatrix}\beta_2\\..\\\beta_5\end{pmatrix}=\begin{pmatrix}0\\..\\0\end{pmatrix}\): The overall significance of the model.
\(HO:\beta_{II}=0\): where \(\beta=\begin{pmatrix}\beta_I=\begin{pmatrix}\beta_1\\\beta_2\end{pmatrix}\\\beta_{II}=\begin{pmatrix}\beta_3\\\beta_4\\\beta_5\end{pmatrix}\end{pmatrix}\). The significance of the regressors \(X_3\), \(X_4\), and \(X_5\) as a whole.

So each of the above hypotheses can be written in the form of (3.62) as follows:

\(\underset{(1\times 5)}{\overset{R}{(0,0,1,0,0)}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(1\times 1)}{\overset{r}{0}}\quad \text{The result is a scaler}\).
\(\underset{(1\times 5)}{\overset{R}{(0,0,0,1,0)}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(1\times 1)}{\overset{r}{1}}\quad \text{The result is a scaler}\).
\(\underset{(1\times 5)}{\overset{R}{(0,1,0,0,1)}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(1\times 1)}{\overset{r}{3}}\quad \text{The result is a scaler}\).
\(\underset{(1\times 5)}{\overset{R}{(0,1,0,-1,0)}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(1\times 1)}{\overset{r}{0}}\quad \text{The result is a scaler}\).
\(\underset{(4\times 5)}{\overset{R}{\begin{pmatrix}0&1&0&0&0\\0&0&1&0&0\\0&0&0&1&0\\0&0&0&0&1\end{pmatrix}}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(4\times 1)}{\overset{r}{\begin{pmatrix}0\\0\\0\\0\end{pmatrix}}}\quad \text{The result is a vector}\).
\(\underset{(3\times 5)}{\overset{R}{\begin{pmatrix}0&0&1&0&0\\0&0&0&1&0\\0&0&0&0&1\end{pmatrix}}}\underset{(5\times 1)}{\overset{\beta}{\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\end{pmatrix}}}=\underset{(3\times 1)}{\overset{r}{\begin{pmatrix}0\\0\\0\end{pmatrix}}}\quad \text{The result is a vector}\).

As we can see, we can not use the formula (3.62) as is since \(\beta\) is usually unknown, but we can replace this unknown vector by \(\widehat\beta\) instead. However, this replacement requires knowing the probability distribution of the new quantity of \(R\widehat\beta\) for enabling the derivation of the suitable statistic tests.

Before demonstrate later on how they are derived, the most popular tests used in practice are the Wald test (W), the Lagrange multiplier test (LM), and the likelihood ratio test (LR). Although they give approximately the same result as in the classical linear regression models, they are different in terms of what type of linear model they use. The W-test uses only the unrestricted model, the LM-test uses the restricted one, and the LR-test uses both of which, the restricted and the unrestricted one.

To well understand the derivation process of those tests, we should first understand what we mean by restricted model.

3.9.3 Restricted model

The only information used for the OLS regression is that contained in the sample. That is, we do not have any other prior information (out of the sample). However, if the investigator possesses some extra information concerning some of the population parameters, then she/he can translate this information into the \(R\beta=r\) and then using it as a constraint along with the objective function of the OLS method (3.3).

Each time we face a function subject to some constraints, we should think first to call the Lagrange function as we did in the section 3.5. So the minimization of the sum squared residuals (3.3) subject to \(R\beta=r\) is:

\[\begin{equation} L(\beta,\lambda)=(y-X\beta)^t(y-X\beta)+2\lambda^t(R\beta-r) \tag{3.63} \end{equation}\]

Since the Lagrange multiplier \(\lambda\) is an arbitrary unknown quantity we use instead \(2\lambda\) for ease of derivation.

Derivatives with respect to \(\beta\) and \(\lambda\) are:

\[\begin{equation} \begin{cases} \frac{\partial L(\beta,\lambda)}{\partial\beta}=-2X^ty+2X^tX\widehat\beta_R+2R^t\widehat\lambda=0...(a)\\ \frac{\partial L(\beta,\lambda)}{\partial\lambda}=2(R\widehat\beta_R-r)...(b) \end{cases} \tag{3.64} \end{equation}\]

Premultiplying each side of \((a)\) by \(R(X^tX)^{-1}\), we get:

\[\begin{align*} R(X^tX)^{-1}R^t\lambda&=R(X^tX)^{-1}X^ty-R(X^tX)^{-1}X^tX\widehat\beta_R\\ &=R\widehat\beta-R\widehat\beta_R \end{align*}\]

Where \(\widehat\beta\) is the classical OLS estimators vector. Using the second equation \((b)\), the estimates of \(\lambda\) will be:

\[\begin{equation} \widehat\lambda=\big[R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r) \tag{3.65} \end{equation}\]

Now substituting this result into \((a)\), we get the estimate \(\widehat\beta_R\):

\[\begin{equation} \begin{split} \widehat\beta_R&=(X^tX)^{-1}X^ty-(X^tX)^{-1}R^t\widehat\lambda\\&=\widehat\beta-(X^tX)^{-1}R^t\big[R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r) \end{split} \tag{3.66} \end{equation}\]

The mean of that estimator is:

\[\begin{equation} E(\widehat\beta_R)=\beta-(X^tX)^{-1}R^t\big[R(X^tX)^{-1}R^t\big]^{-1}(R\beta-r) \tag{3.67} \end{equation}\]

That means that if the constraint is correct \(R\beta=r\), then the restricted estimator is unbiased \(E(\widehat\beta_R)=\beta\). Otherwise, we get a biased estimator that reflects the rejection of the null hypothesis.

The variance of the restricted estimator \(Var(\widehat\beta_R)\) can be derived by combining (3.66) and (3.67) as follows:

\[\begin{align*} \widehat\beta_R-E(\widehat\beta_R)&=(\widehat\beta-\beta)-\overbrace{(X^tX)^{-1}R^t\big[R(X^tX)^{-1}R^t\big]^{-1}R}^{M_R}(\widehat\beta-\beta)\\ &=(I-M_R)(\widehat\beta-\beta) \end{align*}\]

Then the variance will be:

\[\begin{align*} Var(\widehat\beta_R)&=E\big[\big(\widehat\beta_R-E(\widehat\beta_R)\big)\big(\widehat\beta_R-E(\widehat\beta_R)\big)^t\big]\\ &=(I-M_R)\overbrace{E\big[(\widehat\beta-\beta)(\widehat\beta-\beta)^t\big]}^{\sigma^2_{\widehat\beta}=\sigma^2(X^tX)^{-1}}(I-M_R)^t\\ &=\sigma^2(I-M_R)(X^tX)^{-1}(I-M_R)^t \end{align*}\]

Since the matrix \(M_R\) is idempotent (but not symmetric), we can easily find out that:

\[\begin{equation*} (X^tX)^{-1}M^t_R=M_R(X^tX)^{-1}=M_R(X^tX)^{-1}M_R^t \end{equation*}\]

So using this property, the last expression of the restricted variance will be:

\[\begin{equation} Var(\widehat\beta_R)=\sigma^2\bigg[(X^tX)^{-1}-(X^tX)^{-1}R^t\big[R(X^tX)^{-1}R^t\big]^{-1}R(X^tX)^{-1}\bigg] \tag{3.68} \end{equation}\]

Now it remains to prove that this variance is less than the unrestricted one to reflect the efficiency of a model that has fewer regressors.

Using \(Var(\widehat\beta)=\sigma^2(X^tX)^{-1}\) and \(Var(R\widehat\beta)=\sigma^2R(X^tX)^{-1}R^t\) (because \(R\) is non random matrix), the above formula will be simplified as follows:

\[\begin{align*} Var(\widehat\beta_R)&=\sigma^2(X^tX)^{-1}-\sigma^2(X^tX)^{-1}R^t\big[R\sigma^{-2}\sigma^2(X^tX)^{-1}R^t\big]^{-1}R(X^tX)^{-1}\\ &=\overbrace{\sigma^2(X^tX)^{-1}}^{Var(\widehat\beta)}-\overbrace{\sigma^2(X^tX)^{-1}}^{Var(\widehat\beta)}R^t\big[R\overbrace{\sigma^2(X^tX)^{-1}}^{Var(\widehat\beta)}R^t\big]^{-1}R\overbrace{\sigma^2(X^tX)^{-1}}^{Var(\widehat\beta)}\\ &=Var(\widehat\beta)-Var(\widehat\beta)R^t\big[\overbrace{RVar(\widehat\beta)R^t}^{Var(R\widehat\beta)}\big]^{-1}RVar(\widehat\beta)\\ &=Var(\widehat\beta)-Var(\widehat\beta)R^t\big[Var(R\widehat\beta)\big]^{-1}RVar(\widehat\beta) \end{align*}\]

Since \(Var(\widehat\beta)\) and \(Var(R\widehat\beta)\) are positive definite matrices, and the expression \(R^t\big[Var(R\widehat\beta)\big]^{-1}R\) is quadratic form hence also positive definite, then we can conclude that the OLS estimators of the restricted model are efficient provided that the constraints expressed in (3.62) are correct.

In applications, the above variance we cannot compute, since the \(\sigma^2\) is usually unknown, but it can be readily estimated, however, using what is called the constrained maximum log-likelihood. Analogously to the classical unconstrained maximum log-likelihood discussed in section 3.3, the derivatives with respect to \(\sigma^2\), using this method, will be written in terms of the restricted estimators \(\widehat\beta_R\) as follows:

\[\begin{equation*} \frac{\partial l}{\partial \sigma^2}=-\frac{n}{2\sigma^2}+\frac{1}{2}\bigg(\frac{1}{\sigma^2}\bigg)^2(y-X\widehat\beta_R)^t(y-X\widehat\beta_R) \end{equation*}\]

By setting this equation equal to zero, the estimated variance will be given by:

\[\begin{equation} \widehat\sigma_R^2=\frac{1}{n}(y-X\widehat\beta_R) ^t(y-X\widehat\beta_R)=\frac{e^t_Re_R}{n} \tag{3.69} \end{equation}\]

It should be noted here that this estimator is not unbiased since the unbiased one is \(s_R^2=\frac{e_R^te_R}{n-k_R}\) where \(k_R\) is the number of the restricted model parameters. However, with large samples, they tend to be close to each other, so in this case, we say that the estimator is consistent.

3.9.4 Wald test

This test uses the classical unrestricted OLS estimators vector \(\widehat\beta\), Which is already shown in (3.38) that this vector follows the normal distribution. So \(R\widehat\beta\) will be also normally distributed with mean \(E(R\widehat\beta)=R\beta\), and variance \(Var(R\widehat\beta)=\sigma^2R(X^tX)^{-1}R^t\):

\[\begin{align*} R\widehat\beta&\sim \mathrm N\big(R\beta\quad,\quad \sigma^2R(X^tX)^{-1}R^t\big)\\ (R\widehat\beta-R\beta)&\sim N\big(0\quad,\quad \sigma^2R(X^tX)^{-1}R^t\big) \end{align*}\]

Now if the null hypothesis (of the constraints) \(R\beta=r\) is correct, then \(R\beta\) can be replaced by \(r\) hence the last expression will be rewritten:

\[\begin{equation} (R\widehat\beta-r)\sim N\big(0\quad,\quad \sigma^2R(X^tX)^{-1}R^t\big) \tag{3.70} \end{equation}\]

Then by standardizing and squaring the above random variable \((R\widehat\beta-r)\), we get the formula of W-test that follows the Chi-square distribution:

\[\begin{equation} W=(R\widehat\beta-r)^t\big[\sigma^2R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\sim \chi^2_{(q)} \tag{3.71} \end{equation}\]

Large values for this test leads to the rejection of the null hypothesis. That is, the prespecified restrictions are not satisfied.

As we see, this test can not be computed due to the presence of the unknown quantity \(\sigma^2\). However, we can get rid of it by involving the fisher distribution. As shown in 2.11, a fisher variable can be defined as the fraction of two chi-square variables divided by their degrees of freedom. The first one will be the W-test with \(q\) degrees of freedom, and the second one will be that from (3.41) with \((n-k)\) degrees of freedom:

\[\begin{align*} F^c_{(q,n-k)}&=\frac{(R\widehat\beta-r)^t\big[\sigma^2R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\bigg/q}{\frac{e^te}{\sigma^2}\bigg/(n-k)}\\ &=\frac{(R\widehat\beta-r)^t\big[R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\bigg/q}{e^te\bigg/ (n-k)} \end{align*}\]

As we see, \(\sigma^2\) was stripped out from this formula.

Using \(s^2\) in place of \(e^te\bigg/ (n-k)\) for simplification, the test formula will simplify to:

\[\begin{equation} F^c_{(q,n-k)}=(R\widehat\beta-r)^t\big[s^2R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\bigg/q \tag{3.72} \end{equation}\]

There exists another alternative to compute the w-test without passing by fisher distribution, which is simply by replacing the unknown \(\sigma^2\) by its estimated value \(s^2\). however, the resulted distribution is only an approximation of chi-square:

\[\begin{equation*} W=(R\widehat\beta-r)^t\big[s^2R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\approx\chi^2_{(q)} \tag{3.73} \end{equation*}\]

Let us check the second hypothesis \((HO:\beta_4=1)\) in the example 3.5 using that test. This hypothesis is written:

\[\begin{equation*} \underset{(1,5)}{R}\underset{(1,5)}{\widehat\beta}-\underset{(1,1)}{r}=\begin{pmatrix}0&0&0&1&0\end{pmatrix}\begin{pmatrix}\widehat\beta_1\\\widehat\beta_2\\\widehat\beta_3\\\widehat\beta_4\\\widehat\beta_5\end{pmatrix}-1=\widehat\beta_4-1 \end{equation*}\]

Then each piece of the W-test will be written:

\((R\widehat\beta-r)=(\widehat\beta_4-1)=(R\widehat\beta-r)^t\) since the result is a scaler.
\(R(X^tX)^{-1}R^t=\begin{pmatrix}0&0&0&1&0\end{pmatrix}\underbrace{\begin{pmatrix}c_{11}&c_{12}&..&..&..\\..&c_{22}&..&..&c_{25}\\..&..&c_{33}&..&..\\c_{14}&..&..&c_{44}&..\\..&..&..&..&c_{55}\end{pmatrix}}_{(X^tX)^{-1}}\begin{pmatrix}0\\0\\0\\1\\0\end{pmatrix}=c_{44}\)

Substituting these results into (3.73), we get:

\[\begin{align*} F^c_{(q=1,n-k)}&=(R\widehat\beta-r)^t\big[s^2R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)\bigg/q\\ &=\frac{(\widehat\beta_4-1)(c_{44})^{-1}(\widehat\beta_4-1)}{s^2}\\ &=\frac{(\widehat\beta_4-1)^2}{s^2c_{44}} \end{align*}\]

As we know from 2.11 that \(F_{(1,m)}=t^2_{(m)}\), the last above test reduces to a student test that usually used for a single parameter as follows:

\[\begin{equation*} t^c_{(n-k)}=\frac{ |\widehat\beta_4-1|}{s\sqrt{c_{44}}}=\frac{ |\widehat\beta_4-1|}{s_{\widehat\beta_4}} \end{equation*}\]

Where superscript \(c\) stands for computed.

There exist another equivalent way that converts the W-test in (3.72) in terms of the sum squared of errors of both, the restricted and the unrestricted model. Using (3.54), the sums are related as follows:

\[\begin{equation*} e^t_Re_R-e^te=\widehat\beta_{II}^tX^t_{II}M_IX_{II}\widehat\beta_{II} \end{equation*}\]

Using (3.77), we substitute \((X^t_{II}M_IX_{II})^{-1}\) by \(R(X^tX)^{-1}R^t\) and \(\widehat\beta_{II}\) by \(R\widehat\beta-r\), the above expression becomes:

\[\begin{equation*} e^t_Re_R-e^te=(R\widehat\beta-r)^t\big[R(X^tX)^{-1}R^t\big]^{-1}R\widehat\beta-r) \end{equation*}\]

Substituting this result into (3.72):

\[\begin{equation} F_{(q,n-k)}=\frac{(e^t_Re_R-e^te)\big/q}{e^te\big/(n-k)} \tag{3.74} \end{equation}\]

Intuitively, if the restrictions are correct, then we should expect the difference between the two sums to be significant.

Example 3.6 We will use the above data from the exampe 3.2, small_data for R and small_data_py for python with the true model: \(y_t=10+2.6x_{2t}-1.6x_{3t}+4.9x_{4t}+3.6x_{5t}+5.6x_{6t}+\varepsilon_t\).

Suppose that we want to test the following constraints:

\(HO:\begin{pmatrix}\beta_3\\\beta_5\end{pmatrix}=\begin{pmatrix}0\\0\end{pmatrix}\).
\(HO:\begin{pmatrix}\beta_3\\\beta_5\end{pmatrix}=\begin{pmatrix}-1.6\\3.6\end{pmatrix}\)
\(HO:\beta_6-2\times \beta_2=0\)

The first and the last hypotheses are false, but the second one is correct.

They can be written in the form of (3.62) as follows:

\(HO:\begin{pmatrix}0&0&1&0&0&0\\0&0&0&0&1&0\end{pmatrix}\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\\\beta_6\end{pmatrix}=\begin{pmatrix}0\\0\end{pmatrix}\).
\(HO:\begin{pmatrix}0&0&1&0&0&0\\0&0&0&0&1&0\end{pmatrix}\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\\\beta_6\end{pmatrix}=\begin{pmatrix}-1.6\\3.6\end{pmatrix}\).
\(HO:\begin{pmatrix}0&-2&0&0&0&1\end{pmatrix}\begin{pmatrix}\beta_1\\\beta_2\\\beta_3\\\beta_4\\\beta_5\\\beta_6\end{pmatrix}=0\)

In R:

In this example we will use the two R packages lmtest and car that have all the above discussed tests. If we want to test only the significance of some variables, such as the first hypothesis, we should call the function waldtest from the lmtest package. However, for linear combination constraints, the function linearHypothesis from car package.

library(lmtest)
library(car)

# fitting the original model
mod_R_wtest <- lm(y~., data=small_data)

# testing the first hypothesis
# the first argument is the model, and 
# the second is the constraint specified as a formula
W_R <- waldtest(mod_R_wtest, .~.-x3-x5)
W_R

Since we have a high value for the F-test 1027.2 (and small p-value ‘2.2e-16’), we reject the constraint (the null hypothesis) so that the coefficients are significantly different from zero.

By default this test uses the F-test, suitable for finite samples, but it provides also the approximate \(\chi^2\) version for large samples as follows:

waldtest(mod_R_wtest, .~.-x3-x5, test='Chisq')

[out] Wald test
[out] 
[out] Model 1: y ~ x2 + x3 + x4 + x5 + x6
[out] Model 2: y ~ x2 + x4 + x6
[out]   Res.Df Df  Chisq Pr(>Chisq)    
[out] 1     24                         
[out] 2     26 -2 2054.4  < 2.2e-16 ***
[out] ---
[out] Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

For the second and the third hypotheses we will use the function linearHypothesis

# the constraint matrix R for the second hypothesis
hmatrix <- rbind(c(0,0,1,0,0,0), c(0,0,0,0,1,0))

# the vector r
rh <- c(-1.6,3.6)

# since we have small sample we use the F-test which is the default 
linearHypothesis(mod_R_wtest, hypothesis.matrix = hmatrix, rhs=rh, test='F')

[out] 
[out] Linear hypothesis test:
[out] x3 = - 1.6
[out] x5 = 3.6
[out] 
[out] Model 1: restricted model
[out] Model 2: y ~ x2 + x3 + x4 + x5 + x6
[out] 
[out]   Res.Df    RSS Df Sum of Sq      F Pr(>F)
[out] 1     26 170.15                           
[out] 2     24 142.53  2    27.622 2.3257 0.1193

since the p-value 0.1193 is less than the common threshold 0.05, the test did not reject the null hypothesis so that the constraint is retained, which is a correct decision.

# the matrix R for the third hypothesis
hmatrix <- c(0,-2,0,0,0,1)

# the vector r
rh <- 0

# since we have small sample we use the F-test which is the default 
linearHypothesis(mod_R_wtest, hypothesis.matrix = hmatrix, rhs=rh, test='F')

[out] 
[out] Linear hypothesis test:
[out] - 2 x2  + x6 = 0
[out] 
[out] Model 1: restricted model
[out] Model 2: y ~ x2 + x3 + x4 + x5 + x6
[out] 
[out]   Res.Df    RSS Df Sum of Sq      F   Pr(>F)   
[out] 1     25 198.92                                
[out] 2     24 142.53  1    56.391 9.4957 0.005109 **
[out] ---
[out] Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

With the level of 0.05, we should not reject the constraint, but with a higher threshold, say 0.1, we should do. This inconclusive decision is expected since the true combination is \(5.6-2\times2.6=0.4\), which is not far from zero.

In Python:

in python we use the library statsmodels.

import statsmodels.formula.api as smf

# fit the full model
mod_py_wtest = smf.ols('y~x2+x3+x4+x5+x6', data=small_data_py).fit()

# specify the constraint
hypothesis = '(x3=0,x5=0)'

# call the test 
W_py = mod_py_wtest.wald_test(hypothesis)

[out] C:\Users\metal\R_ECON~1\VENV~1\Lib\site-packages\statsmodels\base\model.py:1912: FutureWarning: The behavior of wald_test will change after 0.14 to returning scalar test statistic values. To get the future behavior now, set scalar to True. To silence this message while retaining the legacy behavior, set scalar to False.
[out]   warnings.warn(

W_py

[out] <class 'statsmodels.stats.contrast.ContrastResults'>
[out] <F test: F=array([[1027.21338848]]), p=5.619764980837583e-24, df_denom=24, df_num=2>

Again, the F-value 1027.21338848 is the same as that obtained from R.

By default the test used is F-test, we can switch to the \(\chi^2\) test by setting the argument use_f to False

mod_py_wtest.wald_test(hypothesis, use_f=False)

[out] <class 'statsmodels.stats.contrast.ContrastResults'>
[out] <Wald test (chi2): statistic=[[2054.42677696]], p-value=0.0, df_denom=2>

# testing the second hypothesis
# specify the constraint matrix R and the vector r
R_py = np.array([[0,0,1,0,0,0],[0,0,0,0,1,0]])
r_py = np.array([-1.6, 3.6])

# call the test
mod_py_wtest.wald_test((R_py,r_py))

[out] <class 'statsmodels.stats.contrast.ContrastResults'>
[out] <F test: F=array([[2.32565185]]), p=0.1193444542855956, df_denom=24, df_num=2>

# testing the third hypothesis
mod_py_wtest.wald_test(np.array([0,-2,0,0,0,1]))

[out] <class 'statsmodels.stats.contrast.ContrastResults'>
[out] <F test: F=array([[9.49570748]]), p=0.005109040707911476, df_denom=24, df_num=1>

3.9.5 LM-test

This test is based only on the restricted model, and we can interpret the Lagrange multiplier \(\lambda\) in (3.63) as the level of the contribution of the constraint in the determination of the resulted estimator. Therefore, small values for \(\lambda\) means that the associated constraint has a limited effect on the restricted estimator, and hence this estimator is closer to the unrestricted one, which means that this constraint is more likely to be not satisfied.

To implement this test we use the estimated value \(\widehat\lambda\) expressed in (3.65) under the null hypothesis \(HO:R\beta=r\). Since it is a function of the normally distributed variable \(\widehat\beta\), then it follows the normal distribution with zero mean:

\[\begin{align*} E(\widehat\lambda)&=\big[R(X^tX)^{-1}R^t\big]^{-1}\big(RE(\widehat\beta)-r\big)\\ &=\big[R(X^tX)^{-1}R^t\big]^{-1}\big(\overbrace{R\beta-r}^{HO:R\beta-r=0}\big)\\ &=0 \end{align*}\]

And variance:

\[\begin{align*} Var(\widehat\lambda)&=E(\widehat\lambda\widehat\lambda^t)\\ &=\big[R(X^tX)^{-1}R^t\big]^{-1}\overbrace{E\big[(R\widehat\beta-r)(R\widehat\beta-r)^t\big]}^{Var(R\widehat\beta)}\big[R(X^tX)^{-1}R^t\big]^{-1}\\ &=\big[R(X^tX)^{-1}R^t\big]^{-1}\big[R\overbrace{Var(\widehat\beta)}^{\sigma^2(X^tX)^{-1}}R^t\big]\big[R(X^tX)^{-1}R^t\big]^{-1}\\ &=\sigma^2\big[R(X^tX)^{-1}R^t\big]^{-1}\overbrace{\big[R(X^tX)^{-1}R^t\big]\big[R(X^tX)^{-1}R^t\big]^{-1}}^{I_q}\\ &=\sigma^2\big[R(X^tX)^{-1}R^t\big]^{-1} \end{align*}\]

Now since we have its variance, we can scale the random variable \(\widehat\lambda\) to get its standardized version, then square it to get a new variable that follows chi-square distribution known as LM-test:

\[\begin{equation} LM=\frac{\widehat\lambda^t\big[R(X^tX)^{-1}R^t\big]\widehat\lambda}{\sigma^2}\sim \chi^2_{(q)} \tag{3.75} \end{equation}\]

Again as we did with the previous test, for this test to get computed, \(\sigma^2\) should be replaced by its unbiased estimator \(\widehat\sigma^2=\frac{e^t_Re_R}{n_k+q}\) (or by the consistent estimator \(\widehat\sigma^2=\frac{e^t_Re_R}{n}\) with large samples) which now leads the LM-test to follow approximately the above distribution \(\chi^2_{(q)}\):

\[\begin{equation} LM=\frac{\widehat\lambda^t\big[R(X^tX)^{-1}R^t\big]\widehat\lambda}{\widehat\sigma_R^2}\approx \chi^2_{(q)} \tag{3.76} \end{equation}\]

Small values for this test leads to the rejection of the constraints.

For instance, if we consider the model (3.44) with the constraint \(HO:\beta_{II}=0\) (where \(\beta_{II}\) has \(\mathrm g\) variables), then the null hypothesis can be expressed in terms of \(R\) as follows:

\[\begin{equation*} R=\big(\underset{(\mathrm g, k-\mathrm g)}{0}\underset{(\mathrm g, \mathrm g)}{\mathrm I}\big) \end{equation*}\]

Hence the expression will be:

\[\begin{equation*} R(X^tX)^{-1}R^t=\big(\underset{(\mathrm g, k-\mathrm g)}{0}\underset{(\mathrm g, \mathrm g)}{\mathrm I}\big)\begin{pmatrix}X_I^tX_I&X_I^tX_{II}\\X_{II}^tX_I&X_{II}^tX_{II}\end{pmatrix}\begin{pmatrix}\underset{(k-\mathrm g, \mathrm g)}{0}\\\underset{(\mathrm g, \mathrm g)}{\mathrm I}\end{pmatrix} \end{equation*}\]

Now by making use of the result \(B_{22}\) from 3.7, the last expression will simplify to:

\[\begin{equation} R(X^tX)^{-1}R^t=\big(X_{II}^tM_IX_{II}\big)^{-1} \tag{3.77} \end{equation}\]

Substituting this result into (3.76), the LM-test, in this case, will be given by:

\[\begin{equation} LM=\frac{\widehat\lambda^t\big(X_{II}^tM_IX_{II}\big)^{-1}\widehat\lambda}{\widehat\sigma_R^2}\approx \chi^2_{(\mathrm g)} \tag{3.78} \end{equation}\]

Notice that the number of constraints here is equal to the number of the variables \(q=\mathrm g\).

As we can see, this formula is more complicated since it requires the computation of \(\widehat\lambda\). However, another simple and practical alternative way to get this test computed using what is called the Auxiliary regression.

Let us first simplify the formula of \(\widehat\lambda\) in (3.65) by using the formula of \(\widehat\beta_{II}\) in (3.47), and the result obtained above \(R(X^tX)^{-1}R^t=\big(X_{II}^tM_IX_{II}\big)^{-1}\) as follows:

\[\begin{align} \widehat\lambda&=\bigg[\overbrace{R(X^tX)^{-1}R^t}^{=(X_{II}^tM_IX_{II})^{-1}}\bigg]^{-1}\overbrace{\big(R\widehat\beta-r\big)}^{=(\widehat\beta_{II}-0)}\notag\\ &=(X_{II}^tM_IX_{II})\widehat\beta_{II}\notag\\ &=\overbrace{(X_{II}^tM_IX_{II})(X_{II}^tM_IX_{II})^{-1}}^{=\mathrm I}X_{II}^tM_Iy\notag\\ &=X_{II}^tM_I\overbrace{(X_I\widehat\beta_R+e_R)}^{=y}\notag\\ &=X_{II}^t\overbrace{M_IX_I}^{=0}\widehat\beta_R+X_{II}^t\overbrace{M_Ie_R}^{=e_R}\notag\\ \widehat\lambda&=X_{II}^te_R \tag{3.79} \end{align}\]

Using then this result, the LM-test in (3.78) will be:

\[\begin{equation} LM=\frac{e^t_RX_{II}\big(X_{II}^tM_IX_{II}\big)^{-1}X_{II}^te_R}{\widehat\sigma_R^2}\approx \chi^2_{(\mathrm g)} \tag{3.80} \end{equation}\]

What does this result mean? To figure out the right interpretation of this result, let us start with the restricted model \(y=X_I\widehat\beta_R+e_R\) that assumes that the null hypothesis stated above \(HO:\beta_{II}=0\) is correct. Therefore, if \(HO\) is indeed correct then we expect the errors \(e_R\) to be orthogonal (uncorrelated) with both, the regressors in \(X_I\), and the regressors in \(X_{II}\). For \(X_I\) we used the orthogonality assumption \(e^tX=0\implies e^t_RX_I=0\), but how do we verify that with \(X_{II}\)? So we can check that by regressing \(e_R\) on the full set of the regressors, called auxiliary regression, so if we detect any overall significance effect from the regressors on \(e_R\) then this effect would pass also to the dependent variable \(y\) via the restricted model, and hence we should reject the null hypothesis.

Suppose that the auxiliary regression (with the classical assumptions) is given by:

\[\begin{equation} e_R=X\widehat\gamma+\upsilon \tag{3.81} \end{equation}\]

Similar to (3.9) in which we used the idempotent and symmetric matrix \(H\) 3.3, the fitted values denoted \(\widehat e_R\) will be \(\widehat e_R=He_R\), and hence the explained sum of squares will be:

\[\begin{align*} \widehat e_R^t\widehat e_R&=e_R^tH e_R\\ &=e_R^tX(X^tX)^{-1}X^te_R\\ &=e_R^t(X_I,X_{II})\begin{pmatrix}X_I^tX_I&X^t_IX_{II}\\X^t_{II}X_I&X^t_{II}X_{II}\end{pmatrix}^{-1}\begin{pmatrix}X^t_I\\X^t_{II}\end{pmatrix}e_R\\ &=(e_R^tX_I,e_R^tX_{II})\begin{pmatrix}X_I^tX_I&X^t_IX_{II}\\X^t_{II}X_I&X^t_{II}X_{II}\end{pmatrix}^{-1}\begin{pmatrix}X^t_Ie_R\\X^t_{II}e_R\end{pmatrix} \end{align*}\]

Using the assumption of the orthogonality condition in the restricted model \(e^t_RX_I=X_I^te_R=0\) and the results of 3.7, it follows that:

\[\begin{align} \widehat e_R^t\widehat e_R&=(0,e_R^tX_{II})\begin{pmatrix}A&B\\C&(X^t_{II}M_IX_{II})^{-1}\end{pmatrix}\begin{pmatrix}0\\X^t_{II}e_R\end{pmatrix}\notag\\ \widehat e_R^t\widehat e_R&=e_R^tX_{II}(X^t_{II}M_IX_{II})^{-1}X^t_{II}e_R \tag{3.82} \end{align}\]

Where \(A\), \(B\), and \(C\) are neglected elements since they cancel out. Not surprisingly, this is the numerator of the LM-test in (3.80) that will be now written (Substituting also \(\widehat\sigma^2_R=\frac{e^t_Re_R}{n}\)):

\[\begin{equation} LM=n\frac{\widehat e_R^t\widehat e_R}{e^t_Re_R}=nR^2_{aux} \tag{3.83} \end{equation}\]

Where \(R^2_{aux}\) is the coefficient of determination computed from the above auxiliary regression (3.81).

In summary, To compute LM test, first, we perform the restricted model that includes only the subset of the regressors that satisfy the constraints under the null hypothesis. Then we regress the resulted errors \(e_R\) on the full set of the regressors (auxiliary regression). Finally, we compute the LM-test (3.83), and decide, depending on its value, whether we should reject the null hypothesis or not.

Example 3.7 We will use the same example above to test the first hypothesis with LM-lesl

In R:

For LM-test, we will use the auxiliary regression demonstrated earlier.

# fit the restricted model which has been already
mod_R_restricted <- lm(y~.-x3-x5, data=small_data)

# take the residuals
e_R <- mod_R_restricted$residuals

# regress the residuals on the full set of regressors
aux_R <- lm(e_R ~., data=small_data)

# compute LM-test by '$n*R^2$'
LM_R <- dim(small_data)[1]*summary(aux_R)$r.squared

# compute p-value with number of restrictions q=2
# with significance level of 0.05
p_value_R <- 1-pchisq(LM_R, 2)

paste('LM-test: ',LM_R)
[out] [1] "LM-test:  30"
paste('p-value: ',p_value_R)
[out] [1] "p-value:  3.05902320474338e-07"

The low p-value suggests to reject the restrictions.

In Python:


from scipy import stats
# fit the restricted model
mod_py_restricted = smf.ols('y~x2+x4+x6', data=small_data_py).fit()

# take the residuals
e_py = mod_py_restricted.resid

# fit the residuals on the full set of the regressors
aux_py = smf.ols("e_py~x2+x3+x4+x5+x6", data=small_data_py).fit()

# compute the LM-test
LM_py = small_data_py.shape[0]*aux_py.rsquared

# compute the p-value
p_value_py = 1-stats.chi2.cdf(LM_py, 2)

print(f'LM-test : {LM_py}')
[out] LM-test : 29.653584139704552
print(f'p-value : {p_value_py}')
[out] p-value : 3.6375252754350385e-07

3.9.6 LR-test

This test is based on the maximum likelihood estimation, and it requires both models , the restricted and the unrestricted one. The idea behind it is simple, if the constraint is correct, then we expect the likelihood (or the log likelihood) of the restricted model to be not significantly different from that of the unrestricted one. It is defined as follows:

\[\begin{equation} LR=\frac{max\, l(\text{restricted model})}{max\,l(\text{unrestricted model})} \tag{3.84} \end{equation}\]

Intuitively, small values for this ratio lead to rejecting the null hypothesis (reject the constraints) since the maximum likelihood for the constrained model is smaller compared to the unrestricted one. In practice, a simplified version of this test is used for the multiple linear models as follows:

\[\begin{equation} LR=2l(\widehat\beta)-2l(\widehat\beta_R)\approx\chi^2_{(q)} \tag{3.85} \end{equation}\]

Note that we used the OLS \(\widehat\beta\) instead of the ML \(\beta_{ML}\) since they are the same in the multiple linear models, as shown earlier.

This form shows that this test follows asymptotically chi-square distribution, but Since the proof requires to learn other concepts that we do not yet cover, we leave it in later on chapters.

the log likelihood of the unrestricted model is already computed in the section 3.3:

\[\begin{equation*} l=Log(L)=-\frac{n}{2}log(2\pi)-\frac{n}{2}log(\sigma^2)-\frac{1}{2\sigma^2}(y-x\beta)^t(y-X\beta) \end{equation*}\]

So the maximum of this log likelihood will be written in terms of \(\widehat\sigma^2=\frac{1}{n}(y-X\widehat\beta)^t(y-X\widehat\beta)\) as follows (remember \(\widehat\beta=\beta_{ML}\) ):

\[\begin{align*} l(\widehat\beta)&=-\frac{n}{2}log(2\pi)-\frac{n}{2}log(\widehat\sigma^2)-\frac{1}{2\widehat\sigma^2}\overbrace{(y-X\beta)^t(y-X\beta)}^{=n\widehat\sigma^2}\\ &=-\frac{n}{2}log(2\pi)-\frac{n}{2}log(\widehat\sigma^2)-\frac{n\widehat\sigma^2}{2\widehat\sigma^2}\\ &=-\frac{n}{2}log(2\pi)-\frac{n}{2}log(\widehat\sigma^2)-\frac{n}{2} \end{align*}\]

Similarly, the maximum log likelihood of the restricted model will be:

\[\begin{equation*} l(\widehat\beta_R)=-\frac{n}{2}log(2\pi)-\frac{n}{2}log(\widehat\sigma^2_R)-\frac{n}{2} \end{equation*}\]

Substituting these results back in (3.85), we get:

\[\begin{align} LR&=2l(\widehat\beta)-2l(\widehat\beta_R)\notag\\ &=-nlog(\widehat\sigma^2)+nlog(\widehat\sigma^2_R)\notag\\ &=nlog\bigg(\frac{\widehat\sigma^2_R}{\widehat\sigma^2}\bigg)\notag\\ LR&=nlog\bigg(\frac{e^t_Re_R}{e^te}\bigg) \tag{3.86} \end{align}\]

Example 3.8 Using again the above exampel to test the first hypothesis with LR-test

In R:

For LR-test we will use the R package lmtest

library(lmtest)
# fit the full model
mod_R_lrtest <- lm(y~., data=small_data)

# fit the restricted model
mod_R_restrict <- lm(y~.-x3-x5, data = small_data)
# compute LR-test

LR_R <- lmtest::lrtest(mod_R_lrtest, mod_R_restrict)

LR_R

[out] Likelihood ratio test
[out] 
[out] Model 1: y ~ x2 + x3 + x4 + x5 + x6
[out] Model 2: y ~ (x2 + x3 + x4 + x5 + x6) - x3 - x5
[out]   #Df   LogLik Df  Chisq Pr(>Chisq)    
[out] 1   7  -65.943                         
[out] 2   5 -132.863 -2 133.84  < 2.2e-16 ***
[out] ---
[out] Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

As we see, this test displays the log-likelihood of both models. Since the p-value is extremely small, we can conclude that the constraint should be rejected.

In Python:

# fit the full model
mod_py_lrtest = smf.ols('y~'+('+').join(small_data_py.columns[1:]),data=small_data_py).fit()

# fit the restricted model
mod_py_restrict = smf.ols('y~x2+x4+x6', data=small_data_py).fit()

# compute the LR-test
LR_py = mod_py_lrtest.compare_lr_test(mod_py_restrict)
LR_py

[out] (1685.7580219782335, 0.0, 3.0)

The output is a tuple, in which the first is the computed LR-test and the second is the associated p-value.

3.9.7 W vs LM vs LR

To compare the previous tests to each other, we will use the same estimated variance \(\widehat\sigma^2=\frac{e^te}{n}\) for all tests, which is a consistent estimator of the theoretical variance of the unrestricted model. This means that the following analysis is valid only in large samples. Then, we will rewrite each test in terms of the F-test.

W-test:

The W-test in (3.73) will be rewritten as follows:

\[\begin{align*} W&=\big(R\widehat\beta-r\big)^t\bigg[\widehat\sigma^2R(X^tX)^{-1}R^t\bigg]^{-1}\big(R\widehat\beta-r\big)\\ &=\frac{\big(R\widehat\beta-r\big)^t\bigg[R(X^tX)^{-1}R^t\bigg]^{-1}\big(R\widehat\beta-r\big)}{e^te\big/n}\\ &=\frac{\big(R\widehat\beta-r\big)^t\bigg[R(X^tX)^{-1}R^t\bigg]^{-1}\big(R\widehat\beta-r\big)\bigg/q}{\frac{n-k}{nq}.\underbrace{\frac{e^te}{n-k}}_{s^2}}\\ &=\frac{nq}{n-k}.\frac{\big(R\widehat\beta-r\big)^t\bigg[s^2R(X^tX)^{-1}R^t\bigg]^{-1}\big(R\widehat\beta-r\big)}{q} \end{align*}\]

Using then (3.72), the W-test will be:

\[\begin{equation} W=\frac{nq}{n-k}F_{(q,n-k)} \tag{3.87} \end{equation}\].

LM-test:

Substituting the estimated Lagrange multiplier \(\widehat\lambda\) from (3.65) into (3.76), we get:

\[\begin{align*} LM&=\frac{\widehat\lambda^t\big[R(X^tX)^{-1}R^t\big]\widehat\lambda}{\widehat\sigma_R^2}\\ &=\frac{(R\widehat\beta-r)^t\big[R(X^tX)^{-1}R^t\big]^{-1}\overbrace{\big[R(X^tX)^{-1}R^t\big]\big[R(X^tX)^{-1}R^t\big]^{-1}}^{\mathrm I}(R\widehat\beta-r)}{\widehat\sigma_R^2}\\ &=\frac{(R\widehat\beta-r)^t\big[R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)}{\widehat\sigma_R^2} \end{align*}\]

Using (3.72) again, The numerator will be rewritten as:

\[\begin{equation*} (R\widehat\beta-r)^t\big[R(X^tX)^{-1}R^t\big]^{-1}(R\widehat\beta-r)=qs^2F_{(q,n-k)}=q\frac{e^te}{n-k}F_{(q,n-k)} \end{equation*}\]

Substituting this result back to the above expression, we get:

\[\begin{align*} LM&=\frac{q.\big(\frac{e^te}{n-k}\big).F_{(q,n-k)}}{e^t_Re_R\bigg/n}\\ &=\frac{n}{n-k}.\frac{e^te}{e_R^te_R}.q.F_{(q,n-k)} \end{align*}\]

Using then (3.74), the expression \(\frac{e^te}{e_R^te_R}\) will be rewritten:

\[\begin{align*} F_{(q,n-k)}&=\frac{(e^t_Re_R-e^te)\big/q}{e^te\big/(n-k)}\\ \frac{q}{n-k}F_{(q,n-k)}&=\frac{(e^t_Re_R-e^te)}{e^te}\\ \frac{n-k}{q.F_{(q,n-k)}}&=\frac{e^te}{(e^t_Re_R-e^te)}\\ \frac{n-k}{q.F_{(q,n-k)}}&=\frac{e^te\big/e^t_Re_R}{(1-\frac{e^te}{e^t_Re_R})}\\ \frac{e^te}{e^t_Re_R}&=\frac{n-k}{q.F_{(q,n-k)}+n-k} \end{align*}\]

Substituting back into the LM-test formula, we get:

\[\begin{align*} LM&=\frac{n}{n-k}.\frac{n-k}{q.F+n-k}.q.F\\ &=\frac{nqF}{qF+n-k} \end{align*}\]

Dividing both the numerator and the denominator by \(qF\), the final form of the LM-test in terms of F-test is:

\[\begin{equation} LM=\frac{n}{1+\frac{n-k}{qF_{(n-k)}}} \tag{3.88} \end{equation}\]

LR-test:

By using again (3.72), the LR-test in (3.86) will be rewritten:

\[\begin{align} LR&=nlog\bigg(\frac{e^t_Re_R}{e^te}\bigg)\notag\\ &=nlog\bigg(\frac{e^t_Re_R-e^te}{e^te}+1\bigg)\notag\\ LR&=nlog\bigg(\frac{qF_{(n-k)}}{n-k}+1\bigg) \tag{3.89} \end{align}\]

Now since all the above tests are expressed in terms of the same variable \(F_{(n-k)}\), we can then compare them. To do so, we make use of the following mathematical inequation:

\[\begin{equation*} \frac{x}{1+x}\leqslant log(1+x)\leqslant x\,\text{where}\,x\in]-1,+\infty[ \end{equation*}\]

For our case, if we set \(x=\frac{qF}{n-k}\), then we have:

\[\begin{align} W&=nx\\ LM&=n\bigg(\frac{1}{1+\frac{1}{x}}\bigg)=n\bigg(\frac{x}{1+x}\bigg)\\ LR&=nlog(1+x) \end{align}\]

Dropping \(n\) we get the following result:

\[\begin{equation} LM\leqslant LR \leqslant W \tag{3.90} \end{equation}\]

We should mention here two critical points that must be taken into account before using any of these tests. The first one is that these tests are valid only when used with large samples, so for small samples, they may give opposite results about the significance of the null hypothesis. The second one is that this result is only valid for the multiple linear regression models with the classical assumptions, so with other type of models, further discussion will be addressed in later chapters.

What we can learn from the above result is that W-test tends to reject the null hypothesis more likely than the other tests, and LR-test in turn more likely than the LM-test. That is, the LM-test is the most conservative one.

We can verify that result by comparing the values of each test computed earlier:

Table 3.12: Test results
engine	LM-test	LR-test	W-test
R	30.0000	133.8400	1027.200
python	29.6536	133.8394	1027.213