3.5 Estimation by Gauss-markov method
This method can be considered as the best method that gives the smallest estimator variance under the unbiasedness condition.
Let \(\beta^m\) be the BLUE estimator:
\[\begin{equation} \beta^m=\underset{(k\times n)}{A}y \tag{3.31} \end{equation}\]
Were \(A\) is a fixed \(k\times n\) matrix.
Substituting \(y\) from the true model (3.2) into the above expression, we get:
\[\begin{equation} \beta^m=A(X\beta+\varepsilon)=AX\beta+A\varepsilon \tag{3.32} \end{equation}\]
compute the mean by including the expectation operator:
\[\begin{align*} E(\beta^m)&=AX\beta+A\overbrace{E(\varepsilon)}^{=0}\\ E(\beta^m)&=AX\beta \end{align*}\]
Since then unbiasedness requires \(AX=I\), we get:
\[\begin{equation} \begin{cases} &E(\beta^m)=\beta\\ &\text{s.t}\quad \underset{(k\times n)}{A}\underset{(n\times k)}{X}=I_k \end{cases} \tag{3.33} \end{equation}\]
Using this condition, the equation (3.32) will be rewritten as:
\[\begin{equation} \beta^m=\beta+A\varepsilon \tag{3.34} \end{equation}\]
The variance matrix of \(\beta^m\) will be computed as follows:
\[\begin{align*} Var(\beta^m)&=E\bigg[\big(\beta^m-\beta\big)\big(\beta^m-\beta\big)^t\bigg]\\ &=E\bigg[\big(A\varepsilon\big)\big(A\varepsilon\big)^t\bigg]\\ &=E\big(A\varepsilon\varepsilon^tA^t\big)\\ &=A\underbrace{E(\varepsilon\varepsilon^t)}_{=\sigma^2I_n}A^t \end{align*}\]
\[\begin{equation} Var(\beta^m)=\sigma^2AA^t \end{equation}\]
This matrix holds the individual variances in its main diagonal. In multivariate regression thus, the rule used to find the efficient estimator is that the sum of all its variances must be the minimum compared to other types of estimators. This rule then will be expressed by the trace of the matrix \(AA^t\) as follows:
\[\begin{align*} \begin{cases} &Min\quad Tr\big(AA^t\big)\\ &\text{subject to}\quad AX=I_k \end{cases} \end{align*}\]
Notice that \(\sigma^2\) is not involved (because assumed to be constant), hence does not affect the minimization process.
To solve the above constrained system, we use the Lagrange function as follows:
\[\begin{align} Lg&=Tr\big(AA^t\big)-Tr\lambda \big(AX-I_k\big)\notag\\ &=Tr\big(AA^t\big)-Tr\big(\lambda AX\big)+\big(AA^t\big)+Tr\big(\lambda \big) \tag{3.35} \end{align}\]
Where \(\lambda\) is \(k\times k\) matrix called The coefficient of Lagrange
Definition 3.4 (Properties of trace operator)
- Trace is the sum of the diagonal elements.
- \(Tr(AB)=Tr(BA)\).
- \(Tr(AA^t)=Tr(A^tA)\).
Using the properties of the \(Tr\), some members of the Lagrange function can be reformulated as follows:
\[\begin{equation*} Tr(A^tA)=Tr(AA^t) \end{equation*}\]
\[\begin{equation*} Tr(\lambda AX)=Tr(\lambda AX)^t=Tr(\underbrace{X^t}_{k\times n}\underbrace{A^t\lambda ^t}_{n\times k})=Tr(\underbrace{A^t\lambda ^t}_{n\times k}\underbrace{X^t}_{k\times n}) \end{equation*}\]
reporting now these transformations into the Lagrange function, we get:
\[\begin{align*} Lg&=Tr(A^tA)-Tr(A^t\lambda^tX^t)+Tr(\lambda)\\ &=Tr(A^tA-A^t\lambda^tX^t)+Tr(\lambda) \end{align*}\]
The differentiation equation with respect to the unknown matrix \(A\) will be:
\[\begin{equation*} dLg=Tr(dA^t.A+A^t.dA-dA^t.\lambda^tX^t) \end{equation*}\]
Since \(Tr(dA^t.A)=Tr(A^t.dA)\), the last expression will be simplified to:
\[\begin{equation} dLg=Tr\big[dA^t\big(2A-\lambda^tX^t\big)\big] \end{equation}\]
Therefore, to get the minimum trace for the matrix \(AA^t\), the differentiation equation must be equal to zero:
\[\begin{align} dLg=0&\implies Tr\big[dA^t\big(2A-\lambda^tX^t\big)\big]=0\notag\\ &\implies \big(2A-\lambda^tX^t\big)=0\notag\\ &\implies 2A=\lambda^tX^t \tag{3.36} \end{align}\]
Then post multiplying each side of the expression in the last line by \(X\) and taking into account the condition \(AX=I\), it follows that:
\[\begin{align*} dLg=0 &\implies 2A=\lambda^tX^t\\ &\implies 2\overbrace{AX}^{=I}=\lambda^tX^tX\\ &\implies \lambda^t=2\big(X^tX\big)^{-1} \end{align*}\]
Substituting this result in (3.36), we get:
\[\begin{equation*} 2A=\lambda^tX^t\implies 2A=2\big(X^tX\big)^{-1}X^t \end{equation*}\]
Finally, the matrix \(A\) is equal to:
\[\begin{equation*} A=\big(X^tX\big)^{-1}X^t \end{equation*}\]
Substituting \(A\) in (3.31), we obtain the Gauss-Markov estimator:
\[\begin{equation} \beta^m=Ay=\big(X^tX\big)^{-1}X^ty \end{equation}\]
Which is the same as that of the OLS method.