Integrating Big Data and More with Your Data Warehouse

In 1988 I published the first data warehouse architecture. Its aim was to provide consistent, integrated data to business users in support of cross-enterprise decision making. Quality and consistency were the key drivers; at that time the major issues were that operational / transactional systems were highly inconsistent and direct access to them was discouraged for reasons of performance and security. Business users were happy to get whatever consistent view they could, and, in general, wanted to see a stable representation of the business on a monthly, weekly or occasionally daily basis. This architecture has remained a foundation of business intelligence ever since.

21 years later, in 2009, I introduced Business Integrated Insight (BI²). With emerging needs like near real-time decision making in operational BI and increasing use of non-traditional data coming from Web 2.0 and other sources, this new architecture had to address a far wider scope than the original data warehouse. While consistency and integrity remain important considerations, today’s business needs are far more about instant access to the ever-changing ebb and flow of trends in sales, manufacturing and more. It was becoming clear that a new, over-arching architecture was required to cover all the information, processes and people of the business.

Now, three years later, it’s clear that traditional BI is racing to keep up with developments in big data, data virtualization and the cloud, mobile computing as well as social networking and collaboration. All these topics were incorporated in BI² from the outset. Now, as the technology moves to the mainstream, we can and must to dive deeper in these specific areas. Big data leads clearly to the impossibility of routing all information through an enterprise data warehouse (EDW). But, how will that impact our need for consistency and integrity? I envisage we will move from the old adage of “a single version of the truth” to multiple versions depending on users’ needs, with one particular version that I call core “business information” being the source of truth for external reporting and financial governance needs.

Data virtualization has also become big news in recent years. In many ways, it’s a technology whose time has come. With the explosion of data volumes and varieties, users need ways to combine data on the fly with confidence and performance. Data virtualization addresses these needs and is increasingly overlapping with function we traditionally associate with ETL. The result, data integration, as it’s sometimes called, enables us to envisage a future where data is made available to users as they need it, whether real-time or integrated and historicized.

And, against the background of all this upheaval in data and infrastructure, we also see a new breed of technology-savvy business users moving into positions of power. These so-called millennials are demanding seamless, mobile access to the information they need, as well as the ability to play with it as required. The rule of IT over the data and application resources of the organization is coming to an end. But, that’s not to say that IT has no future role. In fact, I see more of a fully symbiotic partnership between business and IT emerging, a partnership I call the “biz-tech ecosystem”.

My 2012 BI² Seminar in Rome on 11-12 June explores these new directions and provides guidance on their introduction in your existing data warehouse environment. It also introduces the Advanced Information Warehouse, shown above, as the next step on your journey from a traditional data warehouse to comprehensive business integrated insight.

More Read