In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI). We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more that enable teams to be more agile in their approach to DW/BI development.
In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.
There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:
- Initial modelling and planning are high-level, not detailed. This enables disciplined agile teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
- Development proceeds in small, incremental steps in which vertical slices of the solution are developed. Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through. This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
- Development is usage driven, not data driven. The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions. In practice, the structure of the data is a secondary issue at best.
- Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution. This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
- The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle. Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
- Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team. A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.
For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.