10.2 The thread that runs through the book
Two centuries, four traditions and some thirty methods later, it is fair to ask what, if anything, holds all of this together. Three things do: one arbitration that never disappears, one genuine change of criterion, and a short list of facts that no amount of computation has altered.
10.2.1 Two cultures, one trade-off
Underneath the change of vocabulary, the same arbitration returns in every chapter under a different name.
In chapter two it was the choice of the variables to include, with the bias of omitting a relevant one against the variance of adding an irrelevant one. In chapter seven it was the order of an ARIMA, arbitrated by an information criterion. In chapter eight it became explicit as the bias variance decomposition, and it governed the degree of a polynomial, the penalty of a ridge, the depth of a tree, the number of neighbours. In chapter nine it reappeared as the width of a layer, the strength of a dropout and the moment at which to stop training.
Everything else in these two hundred years has changed. This has not.
10.2.2 What the change of culture really was
The honest summary of the passage from chapter seven to chapter eight is not that the models became more complicated. It is that the criterion of truth moved.
In the econometric tradition a model is judged by the plausibility of its assumptions and by tests applied to the sample that produced it. In the learning tradition it is judged by its error on data it has never seen. Neither criterion is superior in the abstract; they answer different questions, and the mistake is to apply one to the other. A model may predict superbly and be useless for policy, because it has learned an association that a change of policy would destroy. A model may be correctly specified and predict badly.
The practical question to ask before choosing a method is therefore not “which is the best model”, but: do I want to know what would happen if I intervened, or do I want to know what will happen if I do not? The first is a question of effect, and the first seven chapters answer it. The second is a question of forecast, and the last two answer it.
| the question asked | the natural tool | chapter |
|---|---|---|
| what is the effect of x on y | linear model, instrumental variables, ECM | 2-5 |
| is this relation stable over time | unit root and cointegration tests | 7 |
| what value will y take next month | ARIMA, SARIMA, or a recurrent network | 7 and 9 |
| which of these observations resemble each other | clustering, then an internal index | 8 |
| which variables actually matter | lasso, or the importance of a forest | 8 |
| will this new observation be a fraud | a classifier and a metric fit for imbalance | 8 |
| what does this image contain | a convolutional network, often pre-trained | 9 |
10.2.3 What has not changed at all
Four things have survived every revolution described above, and they are worth stating plainly because they are what a reader should retain when the techniques have aged.
The data decide. No architecture repairs a badly measured variable, a biased sample or a mislabelled target. The most sophisticated model in this book, applied to poor data, produces a confident wrong answer faster than a simple one.
Correlation is still not causation. A network with a hundred million parameters that predicts perfectly has established an association. Chapter four’s discussion of endogeneity applies to it exactly as it applied to a two variable regression, and the size of the model changes nothing.
Extrapolation remains impossible. Every method here interpolates within the region covered by the training data. Outside it, all of them fail, and the flexible ones fail more spectacularly because nothing constrains them.
Simplicity keeps its value. On tabular data of moderate size, which is the ordinary material of economics, a well specified regression or a gradient boosting will usually beat a network, and will be explainable to the person who has to act on it.