Agile BI (What it is, Why it is)

“I told the project development team what I wanted and all they ever do is hiding for 6 months, never to be heard from and then ultimately they do not deliver what I wanted! Their process doesn’t work!”

Ever heard this statement from customers? Most of us are familiar with not only such statement but a series of similar reactions from customers. Reactions like these enforced BI evangelists to think about the real problem of using waterfall model for BI projects.

This problem lead to the new model for BI projects, called Agile BI.

Agile is all about Evolution. With Agile mode of work, Projects follow iterative development model rather than going to traditional waterfall model. Agile BI is getting fame in market because it simplifies and accelerates the development of business intelligence applications. It is an evolution of traditional BI development methods that continues to evolve as best practices and project experiences accumulate.

Since Agile can mean different things in the overall context of BI, I think it is worthwhile to take a ‘first principles’ approach. Agility in BI has to be evaluated in context of the Agile Manifesto, (http://agilemanifesto.org) which describes a set of values (4) & principles (12) of the agile way of doing things.

To me, the following core tenets of agile methodology make it very relevant for BI systems:

1) Shared vision among teams
2) Frequent releases that makes business sense
3) Relentlessly manage scope
4) Creating a multi-release framework with a master plan
5) Accommodating changes gracefully

Following are few characteristics of an Agile BI:

Iterative, Incremental, Evolutionary – First and foremost Agile Business Intelligence is an iterative, incremental, and evolutionary style of development. We work in short iterations that are generally 1-3 weeks long, and never more than 4 weeks. We build the system in small increments or “chunks” of user-valued functionality. And we evolve the working system by adapting to frequent user feedback.

Feature Driven Development – The goal of each development iteration is the production of user-valued features. What users of BI systems care about is the presentation of, and access to information that either helps them solve a business problem or make better business decisions. Every iteration must produce at least one new user-valued feature in spite of the fact that user features are just the tip of the architectural iceberg that is a BI system.

Production Quality – Each newly developed feature must be fully tested and debugged during the development iteration. Agile development is not about building hollow prototypes, it is about incrementally evolving to the right solution with the best architectural underpinnings. We do this by integrating testing and QA early and continuously into the development process

Barely Sufficient Processes – Traditional styles of BI development are rife with a high degree of ceremony. I’ve worked on many projects that involved elaborate stage gate meetings between stages of development such the transition from requirements analysis to design. These gates are almost always accompanied by a formal document that must be “signed off” as part of the gating process. In spite of this ceremony many BI projects struggle or founder. Agile BI emphasizes a sufficient amount of ceremony to meet the practical needs of the project (and future generations) but nothing more.

Automation, Automation, Automation – The only way to be truly agile is to automate as many routine processes as possible. Agile BI teams seek to automate any process that is done more than once. The more you can automate, the more you can focus on developing user features. Test automation is perhaps the most critical form of automation. If you must test your features and system manually, then guess how often you’re likely to rerun your tests? Test automation enables you to frequently re-validate that everything is still working as expected.

Collaboration – Too often in traditional projects the development team solely bears the burden of ensuring that timelines are met, complete scope is delivered, budgets are managed, and quality is ensured. Agile business intelligence acknowledges that there is a broader project community that shares responsibility for project success. The project community includes the sub-communities of users, business owners, stakeholders, executive sponsors, technical experts, project managers, etc. Frequent collaboration between the technical and user communities is critical to success

Self-Organizing / Self- Managing Teams – Hire the best people, give them the tools and support they need, then stand aside and allow them to be successful. There is a key shift in agile project management style compared to traditional project management. The agile project manager’s role is to enable the team to work their magic; and to facilitate a high degree of collaboration with users and other members of the project community

I have found agile framework especially useful in the BI context where development (enhancements to BI landscape) and support (sustenance of the existing BI infrastructure) happens concurrently.

BI professionals have long been advocates of 90-day increments: Get the results to the users quickly in order to continue building momentum and interest in ongoing project funding. However, incremental isn’t agile no matter how short the increments or how fast the iterations. The agile difference is in participation and interaction–not just building things quickly, but building the right things quickly. The participative model of agile brings with it the promise of breaking away from the “just another report” syndrome that plagues virtually every BI team.

In my view, Agile BI is clearly here to stay and succeed, since most traditional, non-Agile BI platforms and applications will be increasingly challenged to keep pace with a world moving at lightning speed.