On March 21 2017 we ran a webinar entitled Measuring Agile: A Discipline Agile Approach to Metrics. We unfortunately didn’t have enough time to answer all of the great questions that we received so we said that we’d write a blog with the answers. This is it.
We’ve organized the blog into the following sections:
First and foremost, please read our blog entitled 7 “Easy” Steps to Convince Management to Support a Hard Change. This blog has a lot of great advice for getting support from senior management for the changes asked about below.
Q: What is the message to deliver to the board of directors of a company and/or sponsors of projects when it comes to metrics?
The principles section of the webcast made this pretty clear I think. The principles we discussed are:
- There is no easy answer
- Every metric has strengths and weaknesses
- Compete against yourself
- Measure to improve
- You get what you measure
- Measure outcomes at the team level
- Each team needs a unique set of metrics
- Team use metrics to self organize
- Trust but verify
- Adopt common metrics categories across teams
- Don’t manage to the metrics
- Automate wherever possible
- Prefer trends over scalars
- Prefer leading over trailing metrics
- Prefer pull over push
If I had to pick a key message, it’s that you need to be flexible and enable teams to take a context sensitive approach.
Q: How do you address the resistance to ranged estimates? We see a lot of resistence to this with leaders!
Sadly this is all too common. There is a desire for an “exact number” for the cost/schedule for an IT project, the belief being that such precision leads to lower financial risk. This never seems to work out, and in practice it tends to increase overall financial risk. Worse yet it motivates some very ethically questionable management practices such as padding the budget (lying about the cost to get the fixed estimate as close to the upper end of the range as possible), dropping scope late in the lifecycle (lying about what will be delivered and wasting money working on stuff you had no hope of delivering in the first place), or asking for more money when it’s clear you’re not going to deliver it (this is arguably extortion). In the article Estimating on Agile Projects I go into further detail about the need for ranged estimates.
As we discuss in the 7 Easy Steps blog, you need to educate management on the trade-offs involved with their current approach and help the to see that it isn’t working out well for them in practice.
Q: My management definitely wants to know if they are on time and on budget… how should I handle this problem?
This is also another common problem in more traditional organizations, and is sadly a mindset that is often promoted by project management canon. The first thing to do is ask them why “on time and on budget” is important to them, and to continue to explore their goals via an evolutionary “5 why” strategy until you get to the heart of the matter. Usually the real issue is that they want to reduce schedule risk and financial risk but they only know to ask for on time and on budget respectively. Help them to recognize that they can do better than that. For example, a more effective question to ask instead of “Are we on budget?” would be “Are we spending the money wisely?”, the latter question requiring competent governance to truly answer. Similarly, a more effective question to ask that “Are we on schedule?” would be “Are we going to deliver when the functionality is actually needed?”, also a question requiring better governance than many organizations seem to have today.
Q: Executives and senior management wants common metrics period… can you talk to this – how should a change agent (me) handle this?
Sadly another very common problem. This proves to be selfishly motivated in most cases – they want a common set of metrics to make it easy for them to monitor teams (they have less thinking to do when all the dashboards look the same). As we discussed in the webinar, this isn’t realistic because every team is unique, they face a unique situation, and they have a unique set of priorities. Yes, there are commonalities between teams but also differences. For example, it makes sense for teams to measure quality. BUT, surely it’s obvious that a data warehousing team will measure quality different than a mobile app team, which in turn measures quality differently than a team sustaining a mainframe-based system. This is why I promoted the idea of asking teams to address common categories (such as Quality, Stakeholder Satisfaction, Finance, and so on) but to do so in a manner that makes sense for them.
Q: In a transformation is the ‘stakeholder vision’ milestone something that you see ‘could’ be used as a way to guage adoption of the toolkit across an Enterprise?
Stakeholder vision is a milestone that marks the end of the Inception phase. I’m not sure how it could be used to gauge adoption. It could be used to gauge readiness to move forward with the transformation though.
Q: Is there more thoughts you have on transformation metrics?
Context counts. In general, take a context-sensitive approach where you measure what is important to you. I worked through a lightweight approach to Goal-Quality-Metric (GQM) during the webinar and the OKR approach is also good. There is no one single answer.
Q: How do start-up agile projects best survive in a really non-agile organization?
Sadly, they generally don’t. If you go poking around on the web you’ll find that there’s a lot of advice around this sort of issue, and a large portion of it advising you to jump to another organization.
Q: Please throw some light on Accelaration metric. I’d like to implement that in my org.
We have a detailed blog about acceleration.
Q: Velocity, used to calculate acceleration, can only be calculated if velocy is based on the same point system… I don’t agree acceleration factors out the points, the points have different meaning potentially by project – can you talk to this some more?
You’re wrong, Here’s a quick answer:
- Team 1 has a velocity this iteration of 20 and a velocity 5 iterations ago of 15. Velocity is measured in Atari points on this team. Acceleration = (20 Ataris – 15 Ataris)/15 Ataris = 33% over 5 iterations.
- Team 2 has a velocity this iteration of 30 and a velocity 5 iterations ago of 20. Velocity for this team is measured in Nintendo points by this team. Acceleration = (30 Nintendos – 20 Nintendos)/20 Nintendos = 50% over 5 iterations.
- When I divided Atari points by other Atari points the unit of measure, Atari points, disappeared. Similar thing happened to Nintendo points. Hence acceleration is comparable as long as it’s calculated over a similar time period (if not, adjust so that you are dealing with a similar time period).
- Read the Acceleration blog for details.
Q: Is value is only monetary value is $?
No, it doesn’t have to be but often is. You should measure value in units that are important to your stakeholders. Perhaps value may be measured in market share by them, for example.
Q: Excecutive leaders want to measure “team health” using metrics and compare teams – maybe reward or punish based on these metrics – please talk to this.
This is generally recognized as bad practice because as soon as teams realized that they are being judged by management they’re motivated to game the metrics as best they can. In the case of automated metrics that are difficult to game, then perhaps its possible to compare against that (code quality metrics come to mind). But, management will get what they measure. For example, if they judge teams based on how many defects they fix, chances are pretty good that the team will start identifying relatively trivial defects and then “fix” them to make their metrics looks good. However, if they judge teams based on code quality trends (i.e. Is code quality improving?) they will likely get higher quality code in the long run.
As I said in the webinar, the primary usage of metrics should be to provide insights to teams to help them to improve. Monitoring teams, part of your overall governance strategy, should be an important but secondary concern.
Q: What if the teams don’t agree with the metrics established by management?
My advice is for teams to identify the metrics that make sense for the situation that they face via a lightweight GQM approach (or something similar such as OKR). Management may want to guide teams, perhaps by insisting on certain categories of metrics, see the earlier discussion, or even by suggesting metrics being collected by other teams.
If it’s a situation where management is trying to inflict a common metrics strategy across all teams, which is a relatively bad idea as I discussed earlier, then I think that the team should justify why they don’t agree with the metrics prescribed by management. I also hope that they would suggest a more appropriate strategy and that management would listen to them.
Q: What are some of your favorite tools for metrics?
I typically don’t like recommending tools because the answer to this question is context dependent. Tool choice will be driven by:
- What you are hoping to measure
- Your existing platform(s)
- Your organizational preferences around tooling (are you a Microsoft shop, are you willing to pay for commercial tools, are you willing to adopt open source tools, …)
- Your previous experiences around such tooling, if any
So, identify what you situation is then do a bit of research to identify the tools that are right for you.
Q: Is it realistic at scale to have multiple dashboards? +900 teams.
I’ll turn this one around. Is it reasonable to ask teams not to have dashboards simply because your organization is big? Is it reasonable to give up on monitoring teams because your organization is big? Is it reasonable to give up on automation because your organization is big? You see where I’m going with this.