Making predictions statistically usually requires different methods than explaining the past. This simple insight is not usually known among executives and analysts with only basic understanding of statistics. Instead, we often meet executives who insist on using explanatory models to predict the future. They are unfortunately wrong.
A famous example is that it is quite easy to explain what drove stock market growth historically. However, with this knowledge it is still impossible to predict the future market movement.
In general, predictive models don’t benefit from many independent (explanatory) variables. It may come as a surprise that the best predictive models—at least for short-term forecasts—have no independent variables at all. They are instead based on time-series analysis.
Here’s what Canback & Co often uses (but not always, it all depends on the data and situation). We strive to be as close to generally accepted practices as possible.
- We start with a combination of time-series and regression using a so-called ARMAX model
- We use as few independent variables as possible. Sometimes none, always less than four, typically one or two. We do this to achieve parsimony
- We use independent variables that are in turn predictable (there is no point in using a variable like unemployment if it cannot be predicted). In emerging countries, income or household spending and population (by socioeconomic level) are often suitable. GDP is less useful
- We are highly sensitive to the risk of autocorrelation. If autocorrelation exists, models look like they have predictive power with high r-square but it is just a mirage. We would rather have a lower r-square with a good Durbin-Watson than vice versa
- In the time-series component, we take secular trends and seasonality into account
- We test predictive power with MAPE (mean absolute percentage error), not r-square which has little to do with predictive accuracy
Moreover, we do not believe in performing statistical analyses without context. Context includes:
- Conducting market visits (which we do all over the world)
- Painting a high level picture of what is going on from a macro-economic and political perspective, from a distribution and competitor perspective, and from a customer perspective
- Pooling data from many countries. Nothing is more detrimental than to look at a country in isolation
Our findings are aligned with the excellent book on practical, managerial forecasting by Makridakis, Wheelwright, and Hyndman: “Forecasting: Methods and Applications” (especially chapter 11). A must-have for anyone in a managerial position who works only occasionally with statistics.