Chapter 9 Deep learning

The previous chapter ended with algorithms that could draw curved boundaries: a support vector machine with a radial kernel, a forest of trees, a mixture of gaussians. Each of them obtained its flexibility from a device chosen in advance by the statistician, a kernel, a splitting rule, a family of densities. The features themselves were always taken as given.

Deep learning changes that. A neural network stacks simple transformations, each one feeding the next, and it learns the transformations themselves at the same time as the final decision. The first layers build new features out of the raw ones, the following layers build features from those features, and the last layer performs an ordinary regression or classification on the representation that has been constructed. This is the meaning of the word deep: not a more complicated equation, but a composition of many simple ones.

The idea is old, and it failed for a long time for three reasons that have nothing to do with the mathematics: not enough data, not enough computing power, and a few technical obstacles in the training. All three fell between 2006 and 2012, which is why a theory of the nineteen eighties became the dominant technology of the last decade.

This chapter follows the natural order. We start from the linear model of the first chapters and add what is needed to obtain a network, we look closely at how such a network is trained, since this is where all the difficulties live, and we then study the three architectures that made the reputation of the field: convolutional networks for images, recurrent networks for sequences, and adversarial networks for generation.

One methodological remark before we start. The fundamental mechanisms of this chapter, and in particular backpropagation, are written here by hand, in a few lines of R and of numpy. A framework would make them disappear behind a single instruction, which is convenient for working and disastrous for understanding. Only when we reach the large architectures, where writing everything by hand would teach nothing, do we move to a specialized library.