Trends in Business Intelligence

This year, as in every year since the phrase “Business Intelligence” first came to prominence, there have been a rash of predictions about what will happen with the area in 2009. I have linked to and commented on some of these myself on this blog. This article is my take on the area. I should however explain that there is a twist. These are not the trends that I expect to see in 2009, just my thoughts on what ought to be trends in BI over the next few years. I cannot claim to have any prescience about whether they will come to pass, but I will be encouraged if they do.

A more holistic approach to BI in organisations that have already invested
Consolidation of the number of BI platforms with organisations
Operational BI
An increase in the prevalence of BI competency centres
BI teams being jointly owned and managed by business divisions and IT
Data from central warehouses being served to front-end applications
BI data being used to automate some business decisions
Further emphasis on predictive analytics
BI having an increasing role in compliance and risk management
Provision of BI to business partners and/or customers
Incorporation of external data into BI platforms
The most creative users of BI employing it to thrive, even in the current climate

1. A more holistic approach to BI in organisations that have already invested

Often BI solutions have started in a single area. Typically – given its direct link to better managing overall performance – this has been around analysing an organisation’s financial results, though implementations starting in the sales and even operational arenas are also not uncommon. BI systems tend to add more value as they bring in more information, particularly where a high-level phenomenon – e.g. a decrease in expenditure – can be attributed to lower level ones – e.g. greater repeat business (business acquisition being more expensive than repeat business in most industries), plus enhanced productivity, plus reduced staff turnover (resulting in lower recruitment and training costs). To do this, BI needs to widen its scope to take on new data sources and combine them in creative ways. Where BI platforms have already added value, there will be pressure from users to expand them to deliver even more utility and provide more sophisticated insights. This in turn will require BI practitioners to take a more holistic view of the area and to develop roadmaps explaining how new data sources will be brought on stream and what business value will result from this.

2. Operational BI

I think that there is a particular argument for BI to make a greater contribution in the area of operational management, particularly tying this to overall performance. Often BI implementations have focussed on the more stately world of monthly (or weekly) results and steered clear of the daily or hourly demands of operational information. The BI projects that I have seen in the Operational sphere have been more focussed at producing weekly status reports or year-to-date trend analyses. Related to BI platforms expanding to new areas, I see increasing demand for information about the internal effectiveness of an organisation; quickly identifying bottlenecks or areas of inefficiency and tracking efforts to address these. Supporting this type of reporting will often require BI practitioners to rethink their architectures which are often set up to refresh on longer timeframes. This may in turn require the warehouse to support multiple environments with different periodicities.

3. Consolidation of the number of BI platforms with organisations

It is all too common for organisations to have separate BI implementations. Maybe the ERP system comes with some shrink-wrapped cubes, as does the CRM system. Perhaps Finance have invested in some budget analysis tools and different business units have started dabbling in predictive modelling (see 8. below). Maybe certain power users or numerate departments have their own, much cherished databases. Clearly this approach is sub-optimal. If it was ever acceptable to have such an ad hoc approach to the area, the current economic climate means that there is no longer room for a multitude of platforms, each supported by their own ETL, servers, IT teams and (hopefully) training programmes. While transitional costs may make it impractical for some organisations to move to a single platform in the short term, the reductions in licensing, internal support, data centre and training costs that come from standardising BI tools are compelling. This is to say nothing about reducing the level of confusion in users faced with multiple reporting and analysis systems, each with their own terminology, vagaries of functionality and often un-reconciled to each other. Arguments about certain BI tools being best of breed for particular tasks were never wholly convincing, with the breadth of functionality offered by all of the major players today, any of them will be at least good enough for all but the most exacting of applications.

4. An increase in the prevalence of BI competency centres

If this is partly as a result of the previous three trends, it also has some drivers all of its own. Even in multinational organisations with highly devolved structures and local accountability, most of the management information that is needed to run businesses will be relatively consistent from country to country and business unit to business unit. Perhaps the sources will vary, but the business transactions that they support are surprisingly consistent. Many organisations are waking up to this fact and pulling BI provision into the centre. There are obvious benefits to this in terms of cost savings, not reinventing the wheel, increasing the consistency of reporting and enabling corporate roll-ups. However, my own feeling is that the best organisations will supplement a strong central resource with a (probably much smaller) virtual component located close to business needs who can better respond to these in a timely manner, but within an overall information architecture. This hybrid approach offers the best of both worlds.

5. BI teams being jointly owned and managed by business divisions and IT

It is an oft-repeated aphorism that all IT projects are business projects. While clearly true in theory, there are enough counterexamples to suggest that practise is rather different. However, BI projects have often stayed much closer to this maxim than other IT efforts. This is because BI systems serve no purpose unless they are closely entwined with business goals. Natural selection should weed out any non-business-focussed BI projects eventually; even in the sleepiest of organisations. It is likely that this close relationship will deepen. Whilst many aspects of BI are firmly in the IT arena (managing the regular refresh of multiple terabytes of data effectively clearly being one), I see a joint stewardship of the area developing. By this I mean something deeper than business oversight or steering committees. It is also different to business BI liaison managers being appointed – these roles often have the unintended consequence of isolating IT from its customers. What I see emerging in more enlightened organisations is true co-ownership of BI with business and IT management closely collaborating to run the area.

6. Data from central warehouses being served to front-end applications

Of course it is not uncommon for transaction processing systems to have buttons or links that call up a given report. What I am talking about here is a closer coupling, one that does not require users to leave their current system and where information from the warehouse is presented in a seamless manner to users. While such information will have all the BI benefits of being reconciled, consistent and accurate, its provenance will increasingly become invisible to the user (who shouldn’t really have to worry where figures come from so long as they are accurate). This is an area in which web-services have some real potential to leverage the investments that organisations have made in warehouses.

7. BI data being used to automate some business decisions

Stepping a little further along the path from the previous point, even before users of front-end systems have a need to review information about a transaction, it may well be that the information itself has been what determines whether a human is involved or not. Already some organisations perform triage on business transactions. Ones that meet all of a number of requirements get processed automatically. Ones that meet some of them, but not all, may get routed to junior staff, whose role is to determine whether they require further review or can proceed. Finally, those with a major variance to requirements may get sent straight to more senior staff. This approach means that a larger volume of business can be handled and that expensive senior resource is only applied where it is necessary. Of course a prerequisite to this type of approach is having reliable information on which to perform the triage.

8. Further emphasis on predictive analytics

While the best BI implementations encourage users to extrapolate from past figures to estimate future ones, the degree to which the analytical skills of the user plays a part varies from implementation to implementation. Here by analytics I mean the use of larger data sets to identify trends or exceptions, mostly at a portfolio level. Again, having invested in developing data warehouses, it makes sense to utilise the many and sophisticated tools that are available to carry out advanced statistical analysis on these; the aim being to deduce trends that would be beyond even the most skilled of human analysts. A by-product of successful BI implementations is that they often free the more numerate of people in organisations from the burden of repetitive number crunching and allow them to focus on more added-value work such as this.

9. BI having an increasing role in compliance and risk management

Sometimes BI projects may have had their genesis in these areas. However, even when this is not the case a pleasing result of having consolidated much of the organisation’s data in one place is that this then forms a valuable resource for compliance and risk management. Indeed, taking an external perspective, regulators tend to view the existence of an enterprise data warehouse as a sign that an organisation takes managing its risks seriously. Although business managers and risk managers may have slightly different (though hopefully complementary) perspectives and want to answer different types of questions, the data that they need to do this is not that dissimilar. Producing compliance suites from a well-designed warehouse is probably not one of the more taxing BI problems. Again expansion in this area is a further example of point 1. in action.

10. Incorporation of external data into BI platforms

I have spoken above about the remit of BI systems expanding internally and becoming more consistent geographically. A further trend in expansion is to meld internal information with that from external providers. The type of external information would range from industry to industry, but might include market data, information about specific companies or individuals (e.g. credit scores, where the use of these is admissible). For example, in insurance, where I have spent the last twelve years of my career, incorporating information from externally produced flood, or windstorm models is often a priority.

11. Provision of BI to business partners and/or customers

Sometimes one objective of BI implementations is to better understand the relationships with business partners and customers. I have seen this develop to the degree where output from BI systems is mailed to such counterparties. A logical extension of this is to allow such organisations direct access to “their information”. Of course as with any e-commerce initiative, there would have be to strict controls on what is viewed and who can view it, but these are problems that are regularly addressed in other areas of IT and the tools to do this are readily available.

12. The most creative users of BI employing it to thrive, even in the current climate

There have been many articles (including ones written by me) which have spoken about good BI being a great defence in times of economic stress. I would go beyond this and state that the real BI pioneers will take advantage of these capabilities to capture markets from their less well-informed competitors and to steer a course away from areas of business that may bring other less-foresighted organisations down. I look forward to seeing case studies bearing this out appear over the next few years.