2.7 Regression with categorical varaible

Until now, the regressor is supposed to be of numeric type, but in many situations, it can be a factor with some number of levels, such as gender with two levels (Male and Female) or income with low, medium, and high level, etc.

We can encode the character labels (“female”, “male”) with numbers such as 0 and 1 to have a numeric variable called dummy variable, but since it has only two unique values the predicted response variable will have also two unique values as follows:

\[\begin{equation*} \widehat Y_t= \begin{cases} \widehat\beta_0+\widehat\beta_1\overbrace{X_t}^{=0}=\widehat\beta_0 & \text{si}\quad X_t=0,\quad \text{female}\\ \widehat\beta_0+\widehat\beta_1\overbrace{X_t}^{=1}=\widehat\beta_0+\widehat\beta_1 & \text{si}\quad X_t=1,\quad \text{male} \end{cases} \end{equation*}\]

But how can we explain these two values? To answer this question, let us include the expectation operator as follows:

\[\begin{equation} E(\widehat Y_t)=Y_t= \begin{cases} E(\widehat\beta_0)=\beta_0 & \text{si}\quad X_t=0,\quad \text{female}\\ E(\widehat\beta_0)+E(\widehat\beta_1)=\beta_0+\beta_1 & \text{si}\quad X_t=1,\quad \text{male} \end{cases} \tag{2.59} \end{equation}\]

It is clear now that the intercept \(\beta_0\) is the response mean of females , and \(\beta_1\) is the difference between the response mean of males and the response mean of females.

Suppose for simplicity that we want to predict the income of some employees based only on their gender as in the following example.

Example 2.2 we use 500 simulated data where the dependent variable is income, and it is related to the explanatory variable gender such that its mean for males is \(150\) units and for females, \(120\), and the standard deviation is the same for both \(50\)

In R:

# Generate 500 samples with 300 females and 200 males.
gender <- c(rep("Male",length.out=200),rep("Female",length.out=300))

# Convert it to the factor type
gender <- factor(gender)

# Generate the income values to much the mean of each category
set.seed(11)
income <- c(rnorm(200, mean=150,sd=50), rnorm(300,120,50))

# put both variables in a data frame
df_factor <- data.frame(income, gender)

Then we take out 100 samples and fit the model

set.seed(123)
df_sample <- df_factor[sample(NROW(df_factor), 100), ]
model_factor <- lm(income ~ gender, data = df_sample)
tidy(model_factor)

Table 2.9: estimation with factor type in R

term	estimate	std.error	statistic	p.value
(Intercept)	115.01660	6.659729	17.270462	0.0000000
genderMale	22.81763	10.948528	2.084082	0.0397541

Using (2.59), the response mean for females should be equal to 115.01660 and for males 115.01660+22.81763=137.8342. Let us verify these results:

mean_female <-
  mean(df_sample$income[which(df_sample$gender == "Female")])
mean_male <- mean(df_sample$income[which(df_sample$gender == "Male")])

data.frame(mean_female = mean_female,
                        mean_male = mean_male)

#>   mean_female mean_male
#> 1    115.0166  137.8342

As expected, the results are the same.

In Python:

Unlike R, which implicitly handles the categorical variables, in python we should convert those variables to dummies before fitting the model. However, the statsmodels package provides a function that uses the R-like formula to handle the categorical variables like R.

In python, We can generate the same data and fit the model as follows:

from statsmodels.formula.api import ols  
# Generate the factor
factor = ["male"]*200
factor.extend(["female"]*300)

# Generate the income variable
np.random.seed(11)
income = np.random.normal(150, 50, 200).tolist()
income.extend(np.random.normal(120, 50, 300).tolist())

# put the variable in a data frame 
df_p = pd.DataFrame({"income": income, "gender": factor})

# fit the model with the R-style formula 
result_p = ols('income~C(gender)', data=df_p).fit()

# display the results
res_p = result_p.summary2().tables[1]
res_p

Table 2.10: Estimation with factor type in Python

	Coef.	Std.Err.	t	P>|t|	[0.025 ]
Intercept	116.99285	2.829084	41.353609	0	111.43444	122.55126
C(gender)[T.male]	33.39323	4.473175	7.465218	0	24.60461	42.18185