For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters.
For years, some commentators have worried that increasing volumes of data coupled with better and better automated prediction methods would lead to an “end of theory.” What they mean is that the sorts of insights traditional statisticians like to be able to infer from their models of the world (those observations that can be generalized and applied to other problems) are often absent from machine learning algorithms that automatically select hundreds or thousands of parameters. The machine learning methods often work extraordinary well for prediction, but they only give answers—they do not teach lessons.
The machine learning technique they tackled is known as the lasso method, a widely used automated method that ensures models do not get too elaborate. The greatest enemy of predictive analytics (particularly in the “big data” arena) is overfitting, which occurs when a model adheres too closely to a given dataset and becomes less accurate when it is applied to new data; the lasso method helps keep models simpler and more extensible. The problem with the lasso method was that standard significance tests—which help statisticians determine whether a variable is really important or can be thrown out of the model—did not work on it, meaning that it was unable to produce some of the inferential contributions statisticians often demand.
Tibshirani and his colleagues developed a special significance test just for the lasso method (the technical details of which can be found here), and have pointed the way to future research into adding inferential capabilities to other predictive modeling techniques. Although this is only the first step, the promise of more insightful algorithmic methods is exciting. In complex environments such as biological and urban systems, the profusion of variables that might be contributing to a particular effect is enormous, and the value of “big data” prediction paired with generalizable inference may be great as well.