Many executives look for a high R² to assess the quality of a regression analysis, but this is often a mistake. One common example that illustrates why, is the relationship between the stock market and GDP. Regressing the S&P 500 index against GDP, we get an R² of 0.99.
At first glance, this seems to indicate a good fit. However, the result is meaningless*. To understand why, we turn to a standard analysis used by statisticians to assess how good a model is. The graph below indicates a major problem in our model.
A good model should exhibit no systematic trend in these so-called error bars. Here, we see strong clusters of error bars. This is called autocorrelation, and it is a major problem in statistics. Not accounting for autocorrelation will often result in a high R², but this R² is meaningless. Another problem is that both the stock market and GDP share common trends. For example, both are affected by inflation. Co-trending data series will automatically have high R².
Now, let’s try to analyze the S&P 500 and GDP relationship using a better model. We regress the year-over-year percentage change of S&P 500 with that of GDP.
The relationship is random and the R² is close to 0. Yet it is a better model than the one with R²=0.99. We repeat the same error bar analysis we used before and find random errors.
This result is much better than with our high R² model. And we find that GDP does not predict the stock market. This makes sense since we all know that we simply cannot predict the stock market based on GDP, or else we will get rich!
In conclusion, the S&P 500 analysis is a powerful example to show why a high R² can be completely meaningless if other factors such as autocorrelation and underlying trends are not considered. Or as one observer put it: “rainfall is a better predictor of the stock market than GDP”.
Lesson 1: To avoid doing junk statistics, don’t rely on R² to assess how good a
model is. R² only works in certain contexts.
In the next post, we will use an example from the fast-moving consumer goods industry to forecast category demand. Exactly the same problem manifests itself, as it usually does. We will also explain how to interpret predictive models as opposed to explanatory models.
* Except to show that the stock market and GDP grow in sync in the long-term since they are mathematically linked. GDP is roughly 2/3 return on labor, 1/3 return on capital. Return on capital in turn converts into market capitalization (the P/E ratio), and if there were no fluctuations in the labor-capital ratio and in P/E ratios, then R²=1. That is, a mathematical identity.