by Joseph Rickert
by Joseph Rickert
When talking with data scientists and analysts — who are working with large scale data analytics platforms such as Hadoop — about the best way to do some sophisticated modeling task it is not uncommon for someone to say, “We have all of the data. Why not just use it all?” This sort of comment often initially sounds pragmatic and reasonable to almost everyone. After all, wouldn’t a model based on all of the data be better than a model based on a subsample? Well, maybe not — it depends, of course, on the problem at hand as well as time and computational constraints. To illustrate the kinds of challenges that large data sets present, let’s just look at something very simple using the airlines data set from the 2009 ASA challenge.
Here are some of the results for a regression of ArrDelay on CRSDepTime with a random sample of 12,283 records drawn from that data set:
# Coefficients:
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) -0.85885 0.80224 -1.071 0.284
# CRSDepTime 0.56199 0.05564 10.100 2.22e-16
# Multiple R-squared: 0.008238
# Adjusted R-squared: 0.008157
And here are some results from the same model using 120,947,440 records:
#Coefficients:
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) -2.4021635 0.0083532 -287.6 2.22e-16 ***
# CRSDepTime 0.6990404 0.0005826 1199.9 2.22e-16 ***
# Multiple R-squared: 0.01176
# Adjusted R-squared: 0.01176
More data data didn’t yield an obvously better model! I don’t think anyone would really find this to be much of a surprise. We are dealing with a not very good model to begin with. Nevertheless, the example does provide the opportunity to investigate how estimates of the coefficients change with sample size. This next graph shows the coeffients of the slope plotted against sample size with sample sizes ranging from 12,283 to 12,094,709 records. Each regression was done on a random sample that includes about 12,000 points more than the previous one. The graph also shows the standard estimate for the confidence interval for the coefficient at each point in red. Notice that after some initial instability, the coefficient estimates settle down to something close to the value of beta obtained using all of the data.
The rapid approach to the full-data-set value of the coefficient is even more apparent in the following graph that shows the difference between the estimated value of the beta coefficient at each sample and the value obtained using all of the data. The maximum difference from the fourth sample on is 0.07. This is pretty close indeed. In cases like this, if you believed that your samples were representative of the entire data set, working with all of the data to evaluate possible models would be a waste of time an possibly counterproductive.
I am certainly not arguing that one never wants to use all of the data. For one thing, when scoring a model or making predictions the goal is to do something with all of the records. Moreover, in more realistic modeling situations where there are thousands of predictor variables 120M observations might not be enough data to conclude anything. A large model can digest degrees of freedom very quickly and severely limit the ability to make any kind of statistical inference. I do want to argue, however, that with large data sets the ability to work with random samples of the data confers the freedom to examine several models quickly with considerable confidence that results would be decent estimates of what would be obtained in using the full data set.
I did the random sampling and regressions in my little example using functions from Revolution Analytics RevoScaleR package. Initially, all of the data was read from the csv files that comprise the FAA data set into the binary .xdf file format that is used by the RevoScaleR package. Then the random samples were selected by using the rxDataStep function of RevoScaleR. This function was designed to quickly manipulate large data sets. The code below reads a record, draws a random number with a value between 1 and 9999 and assigns it to the variable urns.
rxDataStep(inData = working.file, outFile = working.file, transforms=list(urns = as.integer(runif(.rxNumRows,1,10000))), overwrite=TRUE)
Random samples for each regression were drawn by looping throught the appropriate values of the variables. Notice how the call to R’s runif() function happens within the transforms parameter of rxDataStep. It took about 33 seconds to do the full regression on my laptop which made it feasible to undertake the extravagant number of calculations necessary to do the 1,000 regressions in a few hours after dinner.
I think there are three main take aways from this exercise:
- Lots of data does not necessarily equate to “Big Data”
- For exploratory modeling you want to work in an environment that allows for the rapid prototyping and provides the statistical tools for model evaluation and visualizations. There is no better environment that R for this kind of work, and the Revolution’s distribution of R offers the ability to work with very large samples.
- The ability draw random samples from large data sets is the way to balance accuracy against computational constraints.
To my way of thinking, the single most important capability to implement in any large scale data platform that is going to support sophisticated analytics is the ability to quickly construct, high quality random samples.