We encounter statistical analysis errors all the time. Here are the most common ones we come across when reviewing third party reports, other consulting firms’ deliverables, market researchers, etc.
- No pre-analysis testing of the underlying data. For some reason, most business analysts jump right into the regression or other statistical analyses without testing the underlying data. A regression without rigorous input data testing and adjustments is meaningless. Reject it.
- Running all conceivable independent variables until a few fit. Variable choice should be based on a theory of the case, not a fishing expedition. Reject such analyses.
- Running regressions one variable at a time. It is common to see regressions where independent variables are run one at a time. This is almost always bad since the independent variables tend to influence each other. Such analyses should always be rejected.
- Aiming for the highest R-square and nothing else. This is sometimes good, but usually bad. Ask for the method used and what the interpretation of R-square is in the particular analysis. Note that the concept of R-square does not exist for many methods.
- Using linear regression with time series without taking into account its peculiarities (that is, doing the analysis as if it were a cross-sectional regression). Such superficial analysis usually gives excellent R-square and looks pleasing to the eye when graphed, but is utterly meaningless. Reject it immediately.
Statistical analysis is difficult. There is no way to circumvent this fact. It’s probably fair to say that only PhDs in statistics or similar disciplines know what they are doing, with some laudable exceptions. Yet most businesses don’t have the time and stamina to retain such experts. So how can analysis quality be improved? The best practices we have seen are:
- Have a statistics center of competence that can help the individual analyst. It is a lonely life to be an analyst trying to do good statistical analysis based on vague memories of an undergraduate course, and having superiors who may know even less. A center of competence can help.
- Outsource the analyses to specialists who really know what they are doing. It may look expensive, but getting as correct an answer as the input data allows is usually worth it.
- Ask the right questions. The topics above are a starting point, but there are others. Any managerial statistics book has such questions.