Chapter 8 Machine Learning

The previous chapters were concerned with explaining. We wrote a model, derived the properties of its estimators, and read a coefficient as the effect of one variable on another. This chapter changes the question. We now ask how accurately a quantity can be predicted on observations that were never used to build the model, and we accept, if necessary, that the mechanism producing the prediction is not readable.

That change of objective reorganizes everything. It introduces a vocabulary, supervised and unsupervised learning, training and test sets, overfitting; it replaces the usual diagnostics by metrics computed out of sample; and it brings in a collection of algorithms that no longer look like equations to be estimated but like procedures to be tuned.

The chapter is organized in three movements. We begin with the concepts that all these methods share, and with the way their performance is measured. We then review the supervised algorithms, those that learn from labelled examples, from the regularized regressions to the ensembles of trees. We finish with the unsupervised ones, which are given no answer and must find a structure by themselves, and with the practical steps, missing values, outliers and pre-processing, that come before any model and decide a good part of its success.

Throughout, the same two languages accompany the exposition, and most of the arguments are made with a figure: in this field, seeing what an algorithm does to a data set explains it better than any description.