10.3 What comes next

The remainder of this chapter is a forecast, and the history above should make us modest about it: the people best placed to predict this field declared neural networks dead twice. What follows is therefore a reading of current directions, not a prophecy.

10.3.1 The two cultures are converging

The most substantial development of the last decade is not an architecture, it is the reconciliation of the two traditions that Breiman had opposed. Causal machine learning uses the flexibility of the algorithms of chapter eight to estimate the nuisance parts of a problem, while keeping the identification apparatus of the econometric tradition for the parameter of interest. Double machine learning, introduced around 2018, and the causal forests that estimate heterogeneous treatment effects, are the visible results.

This direction seems the most promising for the reader of this book, because it requires exactly both halves of it: one cannot use these methods without understanding the identification of chapters three to five and the cross validation of chapter eight.

10.3.2 Architectures are becoming general

The Transformer was invented for translation and has since absorbed images, sound, code, protein structures and, more recently, tabular and temporal data. The direction of travel is towards a small number of very general architectures, pre-trained on enormous corpora and adapted to particular tasks by the transfer learning of chapter nine.

For time series in particular, pre-trained forecasting models that produce a forecast without being estimated on the series at hand have appeared. Whether they will displace the ARIMA family on short economic series is genuinely open: the results so far are strong on large collections of series and much less decisive on the single, short, seasonal series that an applied economist usually faces. The reader now has what is needed to test that on their own data instead of believing either side.

10.3.3 Interpretability stops being optional

As these models enter decisions with legal consequences, credit, insurance, hiring, justice, the requirement to explain a decision becomes a constraint rather than a preference. The methods for it, importance measures, partial dependence, local explanations, are developing quickly and are still unsatisfying: they explain the model, not the phenomenon, and two of them can disagree about the same prediction.

This is where the econometric habit of the first chapters keeps a clear advantage, and it is not a nostalgic remark: a coefficient with a standard error and a stated identifying assumption is an object that can be argued about in front of a regulator.

10.3.4 The reproducibility problem

A quieter difficulty threatens the whole field. A result obtained with a network depends on the seed, the version of the library, the hardware and a dozen undocumented choices, and a large share of published results cannot be reproduced exactly. This book was itself written in a way that makes its own numbers reproducible, and doing so revealed how fragile that property is.

The remedies are known and dull: fix the seeds, record the versions, keep the transformations inside a pipeline, and separate the validation set from the test set. None of them is a research topic, and they will matter more than most research topics.

10.3.5 What will not be solved

Some limits are not engineering problems. A model cannot learn an effect that is absent from the data; it cannot compensate for a sample that excludes the population of interest; it cannot settle a causal question that the design does not identify. No amount of parameters or of computation changes any of this, and the temptation to hope otherwise grows with the power of the tools.