Disciplined Agile Strategies for Data Warehouse (DW)/Business Intelligence (BI) Teams

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modellingdatabase refactoringagile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.

In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.

Agile Data Warehousing

There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:

  • Initial modelling and planning are high-level, not detailed.  This enables disciplined agile  teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
  • Development proceeds in small, incremental steps in which vertical slices of the solution are developed.  Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through.  This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
  • Development is usage driven, not data driven.  The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions.  In practice, the structure of the data is a secondary issue at best.
  • Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution.  This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
  • The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle.  Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
  • Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team.  A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.

For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.

Have any Question or Comment?

2 comments on “Disciplined Agile Strategies for Data Warehouse (DW)/Business Intelligence (BI) Teams

Very interesting article. I would be curious, however to see how this model (DAD) would apply with web/eCommerce project (activities per phase) ?


Hi Scott,

Just coming through the conclusion of a DW project that has gone smoothly. I was interested in a couple of finer points. It seems to me that the source-target mapping discussions were a primary activity for us, though weren’t intended to be at the outset. Also, I think we could have had good metadata planning earlier on, involving our SMEs and other end users.


Leave a Reply

Your email address will not be published. Required fields are marked *