Big Data and the New Face of Commerce

Here’s a simple task, let’s say you have an antique that you want to sell on eBay and you need to figure out what it’s worth, what would you do? I’ve sold a few things on eBay and to work out at least an idea of value you can search for similar sold items and develop sort of an average, assuming that there are some recent similar items of course. There are also, depending on the item, online or printed guide books that might help although you would have to make sure the data was reasonably current and that the guide’s methodology was acceptable.

The more data points and in general the more current the data the greater the chance that you could predict the right price point. In fact if you had enough data you might even be able to determine the best time to sell your specific item. In any case data’s the answer and more is definitely better. I read an announcement the other day from WorthPoint, an online resource for the global antiques and collectables community, that explained its new deal to buy eBay antiques data. That data (price, descriptions and images from 150 million transactions) combined with WorthPoint’s other data sources including GoAntiques.com, TIAS and leading auction houses forms the most comprehensive research database for antiques. The data is sourced from a company called Advanced eCommerce Resource Systems, the sole authorized re-licenser of eBay market data.

There are mountains of data created every day; in fact the tally for 2010 was around 1.2 zettabytes (or 1.2 trillion gigabytes) according to a study from IDC commissioned by EMC. To put that in some sort of perspective that would be about 19 billion completely full 64 GB iPads or about 800 trillion copies of the famous novel Atlas Shrugged by Ayn Rand. The report also estimates that about 70% of that data is created by individuals and ranges from transaction data to all manner of socially generated content. Think about the things you do every day, shopping for groceries, bet you swiped your store card for a discount and used a debit or credit card to pay. Maybe you checked in on FourSquare when you got to the store, updated your facebook status, used a groupon coupon and even tweeted about the sale on fresh fish. Let’s just say that the digital footprint you generate every day is significant and very telling directly online and offline in a variety of situations.

That’s half the equation though, all of that data you generated is collected by some organization. Taken individually each data source provides some useful information but if you could aggregate it somehow, the overlap would tell a very complete story…what you like when, daily activities, interests, where you go, what catches your attention and all manner of behavioral clues.

There are three basic flavors of this data, transaction, behavioral (social) and location or contextual. The data can be used, once aggregated and analyzed either individualized and generalized. My opening example of WorthPont’s new antique database is a great example of the generalized use of this data, mostly transaction data in this use case. Generalizing the data can provide everything from pricing trends to brand sentiment or broad public opinions. Brick and mortar stores can use it for merchandising, pricing, planning ad spend, designing loyalty programs, etc.

Individualizing the analysis of the available data can support both online and offline programs including online ad targeting, online and offline offer targeting, online dynamic pricing, individualized marketing programming, etc. Ultimately the inclusion of the behavioral and contextual data can lead to predicting future action, one of the ultimate goals for commerce. Predicting what might capture your interest, what and when you might buy, all of these are gold for merchants.

To some the discussion of this level of data collection and analysis may seem creepy or even an invasion of privacy. The key from a business perspective is to only use data that the individual has given permission to use. Some of the permissions are explicit and some is implicit though. The use of credit cards and store cards, for example, come withy implicit permission to collect and use data. On the other hand, social networks have an extensive set of permissions and controls over personal data (read the fine print though). From a business perspective it’s pretty obvious why there’s some much interest in finding ways to collect, analyze and use all of the data. What’s just as important though is gaining an understanding of what’s in it for the individual? The benefits vary by individual and by the specific use of course but in general the use of this data can make your overall experience with a business more individualized and personal. It can also lead to better and more compelling loyalty programs, sales / offers, coupons, etc. In other words there are some very good personal benefits that are exchanged for giving up some privacy.

The use of data in retail isn’t a new thing, but the amount of data and the ways it can be used is. The addition of the behavioral and contextual data to the existing transaction data is very promising, although many of these systems are pretty new but evolving quickly. Predictive analysis is particularly interesting and promising. Big data and its use is a key component in the new commerce / commerce 2.0. More innovative use is coming as the lines between online and offline blur through the growth of mobile and social commerce. This year should see some innovative new uses and systems to support that use.