A common question that customers ask us is how do you measure productivity on agile teams. If you can easily measure productivity you can easily identify what is working for you in given situations, or what is not working for you, and adjust accordingly. One way to do so is to look at acceleration, which is the change in velocity.
A common metric captured by agile teams is their velocity. Velocity is an agile measure of how much work a team can do during a given iteration. Velocity is typically measured using an arbitrary point system that is unique to a given team or program. For example, my team might estimate that a given work item is two points worth of effort whereas your team might think that it’s seven points of effort, the important thing is that it’s consistent. So if there is another work item requiring similar effort, my team should estimate that it’s two points and your team seven points. With a consistent point system in place, each team can accurately estimate the amount of work that they can do in the current iteration by assuming that they can achieve the same amount of work as last iteration (an XP concept called “yesterday’s weather”). So, if my team delivered 27 points of functionality last iteration we would reasonably assume that all things being equal we can do the same this iteration.
It generally isn’t possible to use velocity as a measure of productivity. You can’t compare the velocity of the two teams because they’re measuring in different units. For example, we have two teams, A and B, each of 5 people and each working on a web site and each having two-week long iterations. Team A reports a velocity of 17 points for their current iteration and team B a velocity of 51 points. They’re both comprised of 5 people, therefore team B must be three times (51/17) as productive as team A. No! Team A is reporting in their point system and B in their point system, so you can’t compare them directly. The traditional strategy, one that is also suggested in the Scaled Agile Framework (SAFe), would be to ask the teams to use the same unit of points. This might be a viable strategy with a small number of teams although with five or more teams it would likely require more effort than it was worth. Regardless of the number of teams that you have it would minimally require some coordination to normalize the units and perhaps even some training and development and support of velocity calculation guidelines. Sounds like unnecessary bureaucracy that I would prefer to avoid. Worse yet, so-called “consistent” measurements such as FPs are anything but consistent because there’s always some sort of fudge factor involved in the calculation process that will vary by individual estimator.
An easier solution exists. Instead of comparing velocities you instead calculate the acceleration of each team. Acceleration is the change in velocity over time. The exact formula for acceleration is:
(NewVelocity – InitialVelocity)/InitialVelocity
For example, consider the reported velocities of each team shown in the table below. Team A’s velocity is increasing over time whereas team B’s velocity is trending downwards. All things being equal, you can assume that team A’s productivity is increasing whereas B’s is decreasing. At the end of iteration 10, if we wanted to calculate the acceleration since the previous iteration (#9), it would be (23-22)/22= 4.5% for Team A and (40-41)/41 = -2.4% for Team B. So Team A improved their productivity 4.5% during iteration 10 and Team decreased their productivity 2.4% that iteration. A better way to calculate acceleration is to look at the difference in velocity between multiple iterations as this will help to smooth out the numbers over time because as you see in the table the velocity will fluctuate naturally over time (something scientists might refer to as noise). Let’s calculate acceleration over 5 iterations instead of just one, in this case comparing the differences in velocity from iteration 6 to iteration 10. For Team A the acceleration would be (23-20)/20 = 15% and for Team B (40-45)/45 = -11.1% during the 5 iteration period, or 3% and -3.7% respectively on average per iteration.
|Iteration||Team A Velocity||Team B
Normalizing Acceleration for Team Size
The calculations that we performed above assumed that everything on the two teams remained the same. That assumption is likely a bit naïve. It could very well be that people joined or left either of those teams, something that would clearly impact the team’s velocity and therefore it’s acceleration. Let’s work through an example. We’ve expanded the first table to include the size of the team each iteration. We’ve also added a column showing the average velocity per person per iteration for each team, calculated by dividing the velocity by the team size for that iteration. Taking the effect of team size into account, the average acceleration between the last five iterations for Team A is (1.9-1.8)/1.8/5 = 1.1% and for Team B is (5-5)/5/5 = 0.
|Iteration||Team A Velocity||Team A Size||Team A Velocity Per Person||Team B
|Team B Size||Team B Velocity Per Person|
Similarly, perhaps there was a holiday during one iteration. When there are ten working days per iteration and you lose one or more of them due to holidays it can have a substantial impact on velocity as well. As a result you may want to take into account the number of working days each iteration in your calculation. You would effectively calculate average acceleration per person per day in this case. Frankly I’m not too worried about that issue as it would affect everyone within your organization in pretty much the same way, and it’s easy to understand why there was a “blip” in the data for that iteration.
What Does Acceleration Tell You?
For how you use acceleration in practice, there are three scenarios to consider:
- Positive acceleration. This is an indication that productivity may be rising on the team, although it does not indicate the cause of that increase.
- Zero acceleration. This is an indication that the team’s productivity is remaining flat, and that perhaps they should consider doing retrospectives regularly and then act on the results from those retrospectives. Better yet they can “dial up” their process improvement efforts by adopting something along the lines of the Disciplined Agile Framework.
- Negative acceleration. If the acceleration is negative then productivity on the team is going down, likely an indicator of quality and/or team work problems.
Of course it’s not wise to govern simply by the numbers, so instead of assuming what is going on we would rather go and talk with the people on the two teams. Doing so you might find out that team A has adopted quality-oriented practices such as continuous integration and static code analysis which team B has not, indicating that you might want to help team A share their learnings with other teams.
This is fairly straightforward to do. For example, assume your acceleration is 0.7%, that there are five people on the team, your annual burdened cost per person is $150,000 (your senior management staff should be able to tell you what this number is), and that you have two week iterations. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person).
Another approach is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (note that you want to avoid comparing iterations near any major holidays). So, if the team velocity the first week of February 2015 was 20 points, the same team’s velocity the first week of February 2016 was 23 points, that’s an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
Advantages of the Acceleration Metric
There are several advantages to using acceleration as an indicator of productivity over traditional techniques such as FP counting:
- It’s easy to calculate. We worked through two common variations earlier, you’ll need to experiment to determine what works for you.
- It is inexpensive. Acceleration is based on information already being collected by the team, their velocity, so there is no extra work to be done by the team. Assuming that the team hasn’t decided to take a #NoEstimates approach
- It is easy to automate. For example, most agile management tools (e.g. VersionOne, Rally, Jira, Microsoft TFS) calculate velocity automatically from their work item list/product backlog and do velocity trend reporting via their team dashboard functionality. This trend reporting is effectively a visual representation of the team’s acceleration (or deceleration as the case may be).
- You can easily adjust for changing team size.
- You can easily monetize this metric.
- It is unitless. The “units” are % change in points per iteration, or % change in points per time period depending on the way that you want to look at it. Because it’s a percentage you can use it as a basis of comparison.
- You apply this across a department. It is fairly straightforward to roll up the acceleration of project teams into an overall acceleration measure for a larger program or portfolio simply by taking a weighted average based on team size. However, this is only applicable to teams that are in a position to report an accurate acceleration (the agile and iterative teams) and of course are willing to do so.
Potential Disadvantages of Acceleration
Of course, nothing is perfect, and there are a few potential disadvantages:
- It can be gamed. Acceleration is derived from velocity which in turn is derived from manually-collected measures, and anything gathered manually can be easily gamed.
- Management must be flexible. For this to be acceptable senior management must be willing to think outside the “traditional metrics box”. Using a non-standard, simple metric to calculate productivity? Preposterous! Directly measuring what you’re truly interested in instead of calculating trends over long periods of time? Doubly preposterous!
- The terminology sounds scientific. Terms such as velocity and acceleration can motivate some of us to start believing that we understand the “laws of IT physics”, something which we doubt very highly that as an industry we understand.
We hope that you’ve found this blog post valuable.