Everywhere you look in tech today you’ll read or hear about big data and all the many uses businesses could get from it. Businesses do certainly need to become more data driven and in fact the business model for the Information Age is something we’ve referred to as “sense and respond.” Moving from the old “make – sell” business model to the new sense and respond model requires data across all your business functions. On the other end of that requirement we’re creating massive amounts of data every second so you’d think that it would be a simple matter to implement the sense and respond model. The problem though. is that having lot’s of data and needing lot’s of data leaves out an important part of this equation…the process of turning mountains of data into actionable information is just not simple.
I’ve written about smart data before
(here), and I still think this is an important part of the key to unlocking the value of the ever growing mountain of data. I’ve started to think though, that there is still something missing from the picture when you think about, and implement ways to use the data to run your business. Part of this relates to the way the big data problem is being addressed and part of it is, I think, using the wrong approach for some of the problems that we need the data to solve.
Big data, when in the big data /non-smart form, is useless, or even more than useless because it can distract businesses and consume a lot of resources for no value return. The problem with the approach, I think, is that big data is the kind of problem technology loves to solve in “big” ways. It needs big resources like really fast servers and multiple kinds of databases and memory and data scientists and analytics engines, not to mention centralized control. Big data is in many ways, the holy grail of IT problems to solve, one that puts lot’s of budget and control right in the IT organization.
Now don’t get me wrong here, there are some major, amazing things that can be done with the data, once it’s turned from it’s big state to the smart state. Smart data is data in context, in the “right hands” and relevant to some issues, activity, problem, etc. Once the data is transformed and processed and delivered to the right person / people in the business that need it to do something of value, it can be an important part of the sense and respond model. Just know that the process of transformation is not so easy or cheap. In the end, if it delivers the right return, or value to the business then great. Smart data is broad though, shows trends and benchmarks and at best can move from reactive to predictive given the right models and context. In other words, it is generally applied broadly, not individually. Here are a few examples of using smart data to return huge business value:
- Banking payment fraud detection and investigation
- Public sector assigning law enforcement based on predicting where / when crimes are likely to happen
- Retail optimizing merchandise placement in stores
- Asset intensive industries monitor and analyze real time machine sensor data to predict and correct failures before they happen
- Insurance claims fraud detection
- Retail making purchase suggestions based on previous transactions and macro buying trends (although that has issues, see the next paragraph)
- Healthcare evidence based medicine
Smart data though, doesn’t provide all the insight that the business needs. Some of the functions are actually tied to individual insight. Part of the beauty of the new world is the capability to be specific and individual. This is very important in marketing, sales, service, and HR, where individualization is a key part of the experiences strategy. Unfortunately many businesses continue to personalize, not individualize. This is the part that many businesses struggle with the most right now. Just think of marketing for example. Most / many of you probably do business with Amazon. I do, and in fact do quite a lot of business with Amazon. Now because of that, I expect Amazon to know my buying habits, interests and some of my needs fairly well.
The problem is, they don’t, which is interesting because they get high praise over their personalization system. They’re applying personalization data to a problem that needs more individualization. The special offers, recommendation, etc. that pop up online and show up in my inbox are ridiculously wrong 99% of the time. Why? Well, first let me say I don’t have any inside knowledge of the Amazon commerce and CRM systems so I may be off base with this analysis, but I suspect that I will be close to the issue.
The problem is that Amazon is using my personal transaction data in conjunction with other smart trend data to build the recommendation and offers. They are looking at the micro experience and rationalizing it to the macro level. Other people “who did this will do that” type of analysis. It’s a reasonable approach but unfortunately, at least in this case, it does not provide the level of individualization that is desired. It is tied to extrapolation of my specific transactions / behaviors into a broad population model of “popular” or the “everyman” average. Now that probably works for commodities like say toothpaste, I’m looking for toothpaste I might be likely to appreciate the “popular” brand recommendation. It does not work on higher end goods though. This is especially true when brand identity plays heavily into the decision.
This idea of mass profiling is interesting, but it has to move beyond the personalization paradigm and stop classifying you into a segment. Segmentation is very popular but is an old concept that misses the mark in a world where my expectation is individualized. It actually is missing the missing the level of relevance that I expect now. With the growth of social web there’s a lot more individual data available and in many cases you have permission to use it. Socialytics is rapidly growing and providing some interesting tools to pull out that individual data. On the other side though, it’s not necessarily getting mapped into the transaction data and other “big data” that would provide a complete picture of my actions and beyond actions, into my behavior and wants / desires. The capability to move beyond personalization and into individualization is critical for building out experiences that meet expectations.
In the next post, I will look at a newish concept called small data. I think that perhaps the big data movement distracts us in some ways from what might be a much bigger revolution around mass democratization of data access and processing. Building out an ecosystem of data and people, or expanding the definition of the enterprise social network (ESN), may be the biggest win for companies.
