2.3 Parameter distributions

Since the estimated model parameters (2.8) \(\widehat\beta_0\) and \(\widehat\beta_1\) are random variables resulting from the randomness of the sample, their probability distributions should be the same as the DGP distribution that is, in most cases, unknown. However, we can assume the normality distribution since this distribution covers most studies in practice.

2.3.1 Probability distributions of variables and parameters in terms of the (unknown) error variance

Formally, The normality assumptions for the errors is expressed by the following short form:

\[\begin{equation*} \varepsilon \sim \mathrm N(0,\sigma^2) \end{equation*}\]

Except for the regressor determined outside the model, everything else is random and related by some way to the normality distribution of the errors.

Definition 2.10 (The moset used density functions in linear rgression)

  • Standard normal density: \[\begin{equation*} X\sim \mathrm N(0,1)\implies f(X)=\frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{1}{2}X^2} \end{equation*}\]

  • Normal density: \[\begin{equation*} X\sim \mathrm N(\mu,\sigma^2)\implies f(X)=\frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{1}{2}\big(\frac{X-\mu}{\sigma}\big)^2} \end{equation*}\]

\[\begin{equation*} if \quad X\sim \mathrm N(\mu,\sigma^2)\implies\bigg(\frac{X-\mu}{\sigma}\bigg)\sim \mathrm N(0,1) \end{equation*}\]

  • Student density:

\[\begin{align*} f(X)&=\frac{\Gamma\bigg(\frac{n+1}{2}\bigg)}{\sqrt{\pi n}\; \Gamma\bigg(\frac{n}{2}\bigg)}\bigg(1+\frac{X^2}{n}\bigg)^{-\frac{n+1}{2}}\implies X\sim \mathrm t(n)\\ Where \quad\mu&=0,\quad \sigma^2=\frac{n}{n-2} \end{align*}\]

  • Fisher density:

\[\begin{align*} f(X)&=\frac{\Gamma\bigg(\frac{m+n}{2}\bigg)}{\Gamma\bigg(\frac{m}{2}\bigg)\Gamma\bigg(\frac{n}{2}\bigg)}\bigg(\frac{m}{n}X\bigg)^{\frac{m}{2}-1}\bigg(1+\frac{m}{n}X\bigg)\frac{m+n}{2}\implies X\sim F(n,m)\\ Where \quad \mu&=\frac{n}{n-2}\quad n\geqslant 3 \quad \sigma^2=\bigg(\frac{n}{n-2}\bigg)^2\frac{2(m+n-2)}{m(n-4)}\quad n\geqslant 5 \end{align*}\]

  • Khi-square density

\[\begin{align*} f(X)&=\frac{1}{2^{\frac{n}{2}}\Gamma\big(\frac{n}{2}\big)}X^{\frac{n}{2}-1}e^{-\frac{X}{2}}\implies X \sim \chi^2(n)\\ Where \quad \mu=n \quad ,\sigma^2=2n \end{align*}\]

Except for the regressor that are determined outside the model, everything else is random and related by some way to the normality distribution of the errors.

Distribution of \(Y_t\):

To characterize the distribution of \(Y_t\), we need the two moments \(E(Y_t)\) and \(V(Y_t)=\sigma_Y^2\). By using the true model (2.7) we have:

\[\begin{align*} Y_t&=\beta_0+\beta_1X_t+\varepsilon_t\\ E(Y_t)&=\beta_0+\beta_1X_t+\underbrace{E(\varepsilon_t)}_{=0}\\ V(Y_t)&=V(\varepsilon_t)\implies \sigma_Y^2=\sigma^2 \end{align*}\]

Using the property of the normal distribution that says, a combination of normally distributed variables is normally distributed. Therefore, since \(Y_t\) is a function of \(\varepsilon_t\), it is then normally distributed with mean \(\beta_0+\beta_1X_t\) and variance \(\sigma^2\):

\[\begin{equation*} Y_t \sim \mathrm N\left(\beta_0+\beta_1X_t\quad,\quad \sigma^2\right) \end{equation*}\]

Notice that this result is based on the fixed regressors assumptions.

Distribution of \(\widehat \beta_1\):

The expression (2.12) will be simplified as:

\[\begin{equation*} \widehat \beta_1-\beta_1=\sum W_t\varepsilon_t\quad where \quad W_t=\frac{\left(X_t-\overline X\right)}{\sum \left(X_t-\overline X\right)^2} \end{equation*}\]

The quantity \(\widehat\beta_1\) is normally distributed since it is a linear combination of \(\varepsilon_t\), so we have:

\[\begin{align} (\widehat\beta_1-\beta_1)&\sim \mathrm N\left(0, \sigma_{\widehat\beta_1}^2\right)\\ \widehat\beta_1&\sim\mathrm N\bigg(\beta_1,\frac{\sigma^2}{\sum \left(X_t-\overline X\right)}\bigg) \tag{2.35} \end{align}\]

The slop \(\widehat \beta_1\) is normally distributed with mean \(\beta_1\) and variance \(\sigma_{\widehat\beta_1}^2=\frac{\sigma^2}{\sum \left(X_t-\overline X\right)}\).

Notice that \(\widehat \beta_1\) can be also written in standardized form which follows the standard distribution:

\[\begin{equation} \frac{\widehat\beta_1-\beta_1}{\sigma_{\widehat\beta_1}}\sim\mathrm N(0,1) \tag{2.36} \end{equation}\]

Distribution of \(\widehat \beta_0\):

The intercept \(\hat\beta_0\) in (2.11) is a linear combination of two normally distributed variables \(\overline Y\) and \(\hat \beta_1\). It is also then normally distributed with mean \(\beta_0\) and variance \(\sigma^2\bigg(\frac{1}{n}+\frac{\overline X^2}{\sum \left(X_t-\overline X\right)^2}\bigg)\), so we have:

\[\begin{equation} \hat\beta_0\sim \mathrm N\bigg(\beta_0,\sigma^2\bigg(\frac{1}{n}+\frac{\overline X^2}{\sum \left(X_t-\overline X\right)^2}\bigg)\bigg) \tag{2.37} \end{equation}\]

Distribution of \(\widehat Y_t\):

In the estimated model (2.8), \(\hat Y_t\) is a linear combination of two normally distributed random variables \(\hat \beta_0\) and \(\hat\beta_1\) with mean:

\[\begin{equation*} E\left(\widehat Y_t\right)=\beta_0+\beta_1X_t \end{equation*}\]

And variance:

\[\begin{align*} V(\widehat Y_t)&=V(\widehat\beta_0)+2X_tcov(\widehat\beta_0,\widehat\beta_1)+X^2_tV(\widehat\beta_1)\\ &=\frac{\sigma^2}{n}+\frac{\sigma^2 \overline X^2}{\sum \left(X_t-\overline X\right)^2}-\frac{2\overline XX_t\sigma^2}{\sum \left(X_t-\overline X\right)^2}+\frac{\sigma^2X_t^2}{\sum \left(X_t-\overline X\right)^2}\\ &=\sigma^2\Bigg[\frac{1}{n}+\frac{X_t^2-2X_t\overline X+\overline X^2}{\sum \left(X_t-\overline X\right)^2}\Bigg]\\ &=\sigma^2\Bigg[\frac{1}{n}+\frac{\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}\Bigg] \end{align*}\]

Then we write:

\[\begin{equation} \widehat Y_t\sim \mathrm N\Bigg(\beta_0+\beta_1X_t,\sigma^2\Bigg[\frac{1}{n}+\frac{\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}\Bigg]\Bigg) \tag{2.38} \end{equation}\]

Distribution of the residuals \(e_t\):

We see from the result (2.20) that the residual \(e_t\) is a combination of the normally distributed variables \(\varepsilon_t\) and \(\widehat \beta_1\), then it is also normally distributed with mean:

\[\begin{align*} E(e_t)&=\underbrace{E(\varepsilon_t-\overline \varepsilon)}_{=0}-\underbrace{E[(\widehat \beta_1-\beta_1)]}_{=0}(X_t-\overline X)\\ &=0 \end{align*}\]

And variance:

\[\begin{align*} V(e_t)&=E(e_t^2)\\ &=E\bigg((\varepsilon_t-\overline \varepsilon)-(\widehat \beta_1-\beta_1)(X_t-\overline X) \bigg)^2\\ &=E\{(\varepsilon_t-\overline \varepsilon)^2+(X_t-\overline X)^2(\widehat \beta_1-\beta_1)^2-2(X_t-\overline X)(\widehat \beta_1-\beta_1)(\varepsilon_t-\overline \varepsilon)\}\\ &=E\left(\varepsilon_t-\overline \varepsilon\right)^2+(X_t-\overline X)^2E\left(\widehat \beta_1-\beta_1\right)^2-2(X_t-\overline X)E\left((\widehat \beta_1-\beta_1)(\varepsilon_t-\overline \varepsilon)\right) \end{align*}\]

Let us treat each member of the right-hand side separately.

Using assumptions 2, 3, and 4, the first member will be:

\[\begin{align*} E\left(\varepsilon_t-\overline \varepsilon\right)^2&=E\left(\varepsilon_t^2-2\varepsilon_t\overline \varepsilon+\overline \varepsilon^2\right)\\ &=E\left(\varepsilon_t^2\right)-2E\Bigg(\varepsilon_t\frac{\sum \varepsilon_t}{n}\Bigg)+E\Bigg(\frac{\sum \varepsilon_t}{n}\Bigg)^2\\ &=E\left(\varepsilon_t^2\right)-2E\Bigg(\frac{\sum \limits_{s=1}^{t-1}\varepsilon_t\varepsilon_s+\varepsilon_t^2+\sum \limits_{s=t+1}^{n}\varepsilon_t\varepsilon_s}{n}\Bigg)+\\ &E\Bigg(\frac{\sum \varepsilon_t^2+\sum \limits_{t<s}\sum \varepsilon_t\varepsilon_s}{n^2}\Bigg)\\ &=\overbrace{E\left(\varepsilon_t^2\right)}^{=\sigma^2}-2\frac{\sum \limits_{s=1}^{t-1}\overbrace{E(\varepsilon_t\varepsilon_s)}^{=0}+\overbrace{E(\varepsilon_t^2)}^{=\sigma^2}+\sum \limits_{s=t+1}^{n}\overbrace{E(\varepsilon_t\varepsilon_s)}^{=0}}{n}+\\ &\frac{\sum \overbrace{E(\varepsilon_t^2)}^{=\sigma^2}+\sum \limits_{t<s}\sum \overbrace{E(\varepsilon_t\varepsilon_s)}{=0}}{n^2}\\ &=\sigma^2-\frac{2\sigma^2}{n}+\frac{n\sigma^2}{n^2}\\ &=\sigma^2\bigg(1-\frac{1}{n}\bigg) \end{align*}\]

For the second member, we use the variance of \(\widehat \beta_1\) (2.16):

\[\begin{equation*} \left(X_t-\overline X\right)^2E\left(\widehat \beta_1-\beta_1\right)=\frac{\sigma^2\left(X_t-\overline X\right)}{\sum \left(X_t-\overline X\right)^2} \end{equation*}\]

Putting aside \(2\left(X_t-\overline X\right)\) for a moment, the last member will be:

\[\begin{align*} E\left((\widehat \beta_1-\beta_1)(\varepsilon_t-\overline \varepsilon)\right)&=E\left((\widehat \beta_1-\beta_1)\varepsilon_t\right)-E\left((\widehat \beta_1-\beta_1)\overline \varepsilon\right)\\ &=E\Bigg(\frac{\sum \left(X_t-\overline X\right)\varepsilon_t}{\sum \left(X_t-\overline X\right)^2}.\varepsilon_t\Bigg)-E\Bigg(\frac{\sum \left(X_t-\overline X\right)\varepsilon_t}{\sum \left(X_t-\overline X\right)^2}.\frac{\sum \varepsilon_t}{n}\Bigg) \end{align*}\]

Again, for simplification, we treat each member of the right-hand side separately, so the first member will be:

\[\begin{align*} &E\Bigg(\frac{\sum \left(X_t-\overline X\right)\varepsilon_t}{\sum \left(X_t-\overline X\right)^2}.\varepsilon_t\Bigg)=\\ &E\Bigg(\frac{\sum \limits_{s=1}^{t-1}\left(X_t-\overline X\right)\varepsilon_t\varepsilon_s+\left(X_t-\overline X\right)\varepsilon_t^2+\sum \limits_{s=t+1}^{n}\left(X_t-\overline X\right)\varepsilon_t\varepsilon_s}{\sum \left(X_t-\overline X\right)^2}\quad\Bigg)=\\ &\frac{\sum \limits_{s=1}^{t-1}\left(X_t-\overline X\right)\overbrace{E(\varepsilon_t\varepsilon_s)}^{=0}+\left(X_t-\overline X\right)\overbrace{E(\varepsilon_t^2)}^{=\sigma^2}+\sum \limits_{s=t+1}^{n}\left(X_t-\overline X\right)\overbrace{E(\varepsilon_t\varepsilon_s)}^{=0}}{\sum \left(X_t-\overline X\right)^2}=\\ &\frac{\sigma^2\left(X_t-\overline X\right)}{\sum \left(X_t-\overline X\right)^2} \end{align*}\]

While the second member will be:

\[\begin{align*} E\Bigg(\frac{\sum \left(X_t-\overline X\right)\varepsilon_t}{\sum \left(X_t-\overline X\right)^2}.\frac{\sum \varepsilon_t}{n}\Bigg)&=E\Bigg(\frac{\sum \left(X_t-\overline X\right)\varepsilon_t^2+\sum \limits_{t\neq s}\sum \left(X_t-\overline X\right)\varepsilon_t\varepsilon_s}{n\sum \left(X_t-\overline X\right)^2}\Bigg)\\ &=\frac{\sum \left(X_t-\overline X\right)\overbrace{E(\varepsilon_t^2)}^{=\sigma^2}+\sum \limits_{t\neq s}\sum \left(X_t-\overline X\right)\overbrace{E(\varepsilon_t\varepsilon_s)}^{=0}}{n\sum \left(X_t-\overline X\right)^2}\\ &=\frac{\sigma^2\overbrace{\sum \left(X_t-\overline X\right)}^{=0}}{n\sum \left(X_t-\overline X\right)^2}\\ &=0 \end{align*}\]

putting all the above results together, The variance of the residual term is:

\[\begin{align*} \sigma_e^2&=\sigma^2\Bigg(1-\frac{1}{n}\Bigg)+\frac{\sigma^2\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}-\frac{2\sigma^2\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}\\ &=\sigma^2\Bigg(1-\frac{1}{n}-\frac{\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}\Bigg) \end{align*}\]

Finally, the distribution of the residual term is:

\[\begin{equation} e_t\sim \mathrm N\Bigg(0,\quad\sigma^2\Bigg(1-\frac{1}{n}-\frac{\left(X_t-\overline X\right)^2}{\sum \left(X_t-\overline X\right)^2}\Bigg) \Bigg) \tag{2.39} \end{equation}\]

We see from that formula that the variance of the residuals \(\sigma_e^2\) is smaller than that of the errors. The reason is that the minimization of \(\sum \varepsilon_t^2\) is reached at the value of \(\sum e_t^2\).

Definition 2.11 (Relationships between distributions)

  • \(\chi^2_{(1)}=\mathrm N^2_{(0,1)}\)

  • \(\chi^2_{(n)}=\sum \limits^n_{i=1}\mathrm N^2_i(0,1)\)

  • \(\chi^2_{(n)}+\chi^2_{(m)}=\chi^2_{(n+m)}\)

  • \(\chi^2_{(n)}-\chi^2_{(m)}=\chi^2_{(n-m)}\)

  • \(t_{(n)}=\frac{\mathrm N_{(0,1)}}{\sqrt{\chi^2_{(n)}/n}}\)

  • \(F_{(n,m)}=\frac{\chi^2_{(n)}/n}{\chi^2_{(m)}/m}\)

  • \(t^2_{(m)}=\frac{\mathrm N^2_{(0,1)}}{\chi^2_{(m)}/m}=\frac{\chi^2_{(1)}/1}{\chi^2_{(m)}/m}=F_{(1,m)}\)

Distribution of \(s^2\):

We start by Squaring and summing (2.20):

\[\begin{equation*} \sum e_t^2=\sum\left(\varepsilon_t-\overline \varepsilon\right)^2-\left(\widehat \beta_1-\beta\right)^2\sum \left(X_t-\overline X\right) \end{equation*}\]

Dividing then each side by \(\sigma^2\) we get:

\[\begin{align} \frac{\sum e_t^2}{\sigma^2}&=\frac{\sum\left(\varepsilon_t-\overline \varepsilon\right)^2}{\sigma^2}-\frac{\left(\widehat \beta_1-\beta\right)^2\sum \left(X_t-\overline X\right)}{\sigma^2}\notag\\ &=\frac{\sum\varepsilon_t^2-n\overline \varepsilon^2}{\sigma^2}-\frac{\left(\widehat \beta_1-\beta\right)^2\sum \left(X_t-\overline X\right)}{\sigma^2}\notag\\ &=\frac{\sum\varepsilon_t^2}{\sigma^2}-\frac{\overline \varepsilon^2}{\underbrace{\sigma^2/n}_{=V(\overline\varepsilon)}}-\frac{\left(\widehat \beta_1-\beta\right)^2}{\sigma^2/\sum \left(X_t-\overline X\right)}\notag\\ &=\frac{\sum\varepsilon_t^2}{\sigma^2}-\frac{\overline \varepsilon^2}{\sigma_{\overline \varepsilon}^2}-\frac{\left(\widehat \beta_1-\beta\right)^2}{\sigma^2_{\widehat\beta_1}}\notag\\ &=\chi^2(n)-\chi^2(1)-\chi^2(1)\notag\\ &=\chi^2(n-2) \tag{2.40} \end{align}\]

Using then the formula (2.22), we obtain:

\[\begin{align*} s^2=\frac{\sum e_t^2}{n-2}&\implies\frac{s^2}{\sigma^2}=\frac{\sum e_t^2}{(n-2)\sigma^2}\\ &\implies(n-2)\frac{s^2}{\sigma^2}=\frac{\sum e_t^2}{\sigma^2}=\chi^2(n-2) \end{align*}\]

That is:

\[\begin{equation} (n-2)\frac{s^2}{\sigma^2}\sim \chi^2(n-2) \tag{2.41} \end{equation}\]

2.3.2 Probability distributions of variables and parameters in terms of the (known) residual variance

Since all the distributions defined above are a function of the variance of disturbances \(\sigma^2\) usually unknown, they cannot be used then in practice. However, those distributions still useful to derive the new distributions after using the estimated variance of \(s^2\) instead.

Distribution of \(\widehat\beta_1\):

Replacing \(\sigma_{\widehat\beta_1}\) by its estimated version \(s_{\widehat\beta_1}\), the new standardized expression of the slop \(\frac{\widehat\beta_1-\beta_1}{s_{\widehat\beta_1}}\) may have a different distribution than that when uses \(\sigma_{\widehat\beta_1}\). So it can be rewritten as follows:

\[\begin{equation*} \frac{\widehat\beta_1-\beta_1}{s_{\widehat\beta_1}}=\frac{\widehat\beta_1-\beta_1\bigg/\sigma_{\widehat\beta_1}}{s_{\widehat\beta_1}\bigg/\sigma_{\widehat\beta_1}} \end{equation*}\]

by first simplifying the denominator as follows:

\[\begin{equation*} \frac{s_{\widehat\beta_1}}{\sigma_{\widehat\beta_1}}=\frac{s\bigg/\sqrt{\sum \left(X_t-\overline X\right)^2}}{\sigma\bigg/\sqrt{\sum \left(X_t-\overline X\right)^2}}=\frac{s}{\sigma} \end{equation*}\]

Then using the standardized expression (2.36), the distribution (2.41), and the correspondence between student and the standard distribution 2.11, we get:

\[\begin{align} \frac{\widehat\beta_1-\beta_1}{s_{\widehat\beta_1}}&=\frac{\widehat\beta_1-\beta_1\bigg/\sigma_{\widehat\beta_1}}{\sqrt{(n-2)s^2_{\widehat\beta_1}\bigg/(n-2)\sigma^2_{\widehat\beta_1}}}\\ &=\frac{\mathrm N(0,1)}{\sqrt{\chi^2(n-2)\bigg/(n-2)}}\\ &=t(n-2) \tag{2.42} \end{align}\]

This quantity will be used as a statistic test for the slop, denoted \(t^c=\frac{\widehat\beta_1-\beta_1}{s_{\widehat\beta_1}}\) where \(c\) means computed from the sample at hand.

Distribution of \(\widehat \beta_0\):

Following the same steps above, the intercept distribution in a standardize form is:

\[\begin{equation} \frac{\widehat\beta_0-\beta_0}{s_{\widehat\beta_0}}=t(n-2) \tag{2.43} \end{equation}\]