Optimization is the application of algorithms to sets of data to guide executives and managers in making the best decisions. It’s a trending topic because using optimization technologies and techniques to better manage a variety of day-to-day business issues is becoming easier. I expect optimization, once the preserve of data scientists and operations research specialists will become mainstream in general purpose business analytics over the next five years.
Optimization is the application of algorithms to sets of data to guide executives and managers in making the best decisions. It’s a trending topic because using optimization technologies and techniques to better manage a variety of day-to-day business issues is becoming easier. I expect optimization, once the preserve of data scientists and operations research specialists will become mainstream in general purpose business analytics over the next five years.
Optimization was first adopted by businesses in the middle of the 20th century, aided by the introduction of digital computers. The first technique that gained broad for a few specific purposes was linear programming, one of the most basic optimization methods. Linear programming enables analysts to quickly determine how to achieve the best outcome (such as maximum unit volume or lowest cost) in a given situation. They do so using a mathematical model that captures the key variables that go into the decision and any constraints that may affect that decision. A food processor, for instance, may use three types of cooking oil to make a product. To maximize its profit, the company needs to determine the exact proportions of the three oils that result in the lowest production cost. However, it can’t just choose the cheapest of the three in every case because for flavor and shelf-life requirements there’s a limit to the maximum percentage of each oil that it can use. Linear programming using the simplex algorithm quickly solves the problem.
As computing capabilities became increasingly affordable, companies could use more complex algorithms to handle ever more difficult optimization problems. For instance, the airline industry used it to determine how best to route aircraft between two cities and to staff flight crews. Not only can software
We’re now on the cusp of “democratizing” what I call optimization analytics. Big data technologies are making it feasible and affordable for even midsize companies to work with much larger data sets than they have been able to in the past. Our benchmark research on big data analytics finds that about half of participating companies already use analytics with big data. This is partly the result of more powerful and affordable data processing resources but also because companies have invested in systems to automate many functions. The rich data sets created by these business applications provide corporations with the raw material for analysis. This data has the potential to enable businesses to make more intelligent decisions. From a practical standpoint, though, the value of these large data sets can only be realized by moving optimization analytics out of the exclusive realm of data scientists and into the hands of business analysts. These analysts are the ones who have a sufficient understanding of the business and the subtleties of the data to find useful and repeatable optimization opportunities. Three-fourths of companies in our research said that they need these business skills (“domain expertise”) to use big data analytics successfully.
Another reason why companies will find it attractive to apply optimization analytics broadly is that the results of applying optimization routines may be superior to using common rules of thumb or relying on instinct and experience. One of the most important lessons for executives about optimization analytics is that optimal solutions are sometimes (but – crucially – not always) counterintuitive to established norms. For instance, in markdown management, retailers often have found that smaller, more frequent price reductions maximize profits and produce a considerable improvement in sales over the end-of-the-season price slashing that was once considered to be the best practice. In financial services, charging your best customers more for loans and other services turns out to be the optimal choice for the bottom line of financial institutions. Another important insight from our collective experience with optimization is that while the value of these analytics as realized in a single event or transaction may be small, it can have a measurable impact on profitability and competitiveness when applied broadly in a business.
At this point optimization analytics is in a dual mode. On the one hand, there are proven examples of the narrow application of optimization such as those mentioned above. On the other, bringing optimization analytics to the masses is only beginning. Some vendors have made progress in simplifying their analytics, but mainstream products are only on the horizon. It’s also important to recognize that, as with past breakthroughs in information technology, there are bound to be more duds than success stories in initial attempts at using optimization analytics. Experience suggests that a small number of companies that have strong analytical skills and a rigorous approach to managing company data will prove to be the leaders in finding profitable opportunities for applying optimization technologies and techniques. Others will do well to find these examples and consider how to apply them to their own organizations.