When Should You Assign Points to Defects?


A common question that we get from teams who are new to agile is whether you should assign points (sizes) to defects.  From a Disciplined Agile point of view we know that the answer will vary depending on the context of the situation, or in other words “it depends.”   The really disciplined answer though is to say on what it depends, which is what we explore in this blog.

Rod Bray, a senior agile coach at Scott Ambler + Associates, suggests this principle:

Don’t reward people for shoddy work

An implication of this is that if a team is injecting defects, they’re producing shoddy work, then it makes sense that their velocity suffers when they have to spend time fixing those defects.  The decrease in velocity is a visible trigger to investigate, and address, the cause of the productivity loss.  But what if the “shoddy work” wasn’t caused by your team?  What if the “shoddy work” was previously acceptable to your stakeholders?  Hmmm… sounds like the context of the situation counts in this case.  The following flowchart works through common thinking around when to size a defect.  Note that this is just our suggestion, your team could choose to do something different.  Let’s work through common scenarios to understand whether defects should be sized or not.  These scenarios are:

  1. Is it existing technical debt?
  2. The defect is injected and found by the team
  3. The defect is found by independent testing
  4. The defect is found after the solution is released

 

When do you size defects

 

Do you size the repair of existing technical debt?

The quick answer is sometimes.  For the sake of discussion we’ll assume that this technical debt has existed for a long time, potentially years.  From a purist point of view technical debt may be injected at any time (this is obviously true) and as a result may only be seconds old.  In the case of technical debt that is very new, refer to the logic below.

The key issue is whether fixing the defect is going to be a substantial effort. This of course is subjective.  Is the work substantial if it’s less than an hour of effort?  Several hours?  Several days?  Several weeks?  This is a bar that your team will need to set.  Something that will impact your schedule is likely to be substantial (and something your Product Owner is likely going to want to prioritize so that it can be planned for appropriately).  Clearly a defect that takes several days or more to fix is substantial and one that takes less than an hour is likely not.  Something that takes several hours or a day or so is likely something you need to think about.

 

Do you size defects injected and by found by the team?

No, the exception being technical debt (see above).  This falls under the principle of not rewarding the team for shoddy work.

 

Do you size defects found by independent testing?

Some teams, particularly those working in regulatory environments or working in complex environments, may be supported by an independent test team.  An overview of the independent testing strategy is presented below.  With the exception of the defect being found in existing technical debt (see above), the defect should not be sized.  Once again, the principle described earlier holds.  Of course if you don’t have an independent test team supporting your team then obviously you can ignore this question.

Independent agile testing

 

Do you size defects found in production?

The answer is sometimes.  As you can see in the high-level lifecycle diagram below, the DA toolkit explicitly recognizes that change requests are often identified by end users of an existing solution.  These change requests are effectively new requirements that should be prioritized and worked on appropriately.  Many organizations will choose to distinguish between two types of change requests, defects and enhancements, so that they may be treated differently.  The issue is that defects are often tracked and, if the team is smart, they are analyzed to determine how they got past the team in the first place.  Additionally you may find that defects and enhancement requests are charged for differently (a topic for a future blog).

Disciplined Agile Lifecycle - High Level System

If you don’t distinguish between defects and enhancement requests then there’s nothing to do, you simply treat the change request as a new requirement and size it like you normally would.  But if you do distinguish between types then you need to think about it a bit.  If the defect is found during a warranty period then it shouldn’t be charged for. Sometimes, particularly when work is being outsourced, there is a warranty period of several weeks or even months after something is released where the development team is expected to fix defects for free.  In this case you likely wouldn’t size the work in line with the principle described earlier.  Once you’re out of the warranty period then it’s likely you want to assign points to it: The functionality in question passed testing and was accepted by your stakeholders, so they in effect have taken responsibility for it.

To summarize, the answer to the question “Should we assign points to a defect?” is “it depends.”  In this blog we explored in detail why this is the case and described when and when you wouldn’t want to assign points.

I’d like to thank Kristen Morton, Rod Bray, and Glen Little for their insights which they contributed to this blog.

Have any Question or Comment?

8 comments on “When Should You Assign Points to Defects?

Great points Scott. I am asked this question a lot and I love this response.

Reply
Valentin Tudor Mocanu

Sometime we want to evaluate defects, but is almost impossible. One of the main problems is to keep the defects in the “sizeable” area and/or keep the development in a quick fixing capability area. If we have too often these kind problems, with a product or product area, we are already in a zone where defects repairing will affect any kind of planning and make the quality questionable by default. In my experience these are one of the worse possible process smells.

In such cases, we should think how to take necessary measures to bring back the product in the track of manageable quality. But we should not loose this track from the first place.

Reply

Hi Scott! I think an important question is how the Defect “points” are going to be used.
– Will they be counted toward Velocity? And is velocity supposed to reflect work capacity (of any kind)? Or only the velocity of value-demand work?
– Are the defects in question going to be part of Sprint planning? Or do you always fix any and all defects within the sprint, before starting work on the next highest valued story?

There is definitely a desire to be able to account for (even plan for) time spent working on defects and other non-value-add work. (Some even wonder if defects should be estimated using negative points)

I think the original intent is for velocity to apply to amount of valued-work (in story points) to apply to planning and progress-tracking for a sprint. In which case, amount of capacity spent working on defects should not contribute to planned nor actual velocity.

Then there is the question of whether to assign points to defects is essentially a different way of deciding if it is really a “bug” or a “feature” for the purposes of planning (or “change-control” of a “project baseline” in traditional waterfall). Let’s certainly hope not!

I think the bigger issue is how to track/account for “friction” or “backwash” (e.g. “undertow”) against the cumulative flow of value-delivery). The initial/simple way is to say defects and other “failure demand” don’t count toward velocity and hence don’t get to have points. But then at some point a team evolves beyond that simplistic view and perceives a need to have more insight about impact on value (e.g., “cost of delay”) of non-value-added work.

So what if defect sizing (“points”) are used, but aren’t counted as part of velocity, and instead as part of something else? (like friction/backwash, or “cost of delay”)

Reply

Brad, you make very good points. I think that the key issues are that you don’t want defects to count towards your value-add velocity yet there is need to track the loss/friction/undertow of quality problems.

In the end, I would go back to basics and ask the questions – How are you going to use this measure (the defect points) to improve your decision making? What will it cost you to gather this measure? Is the decision-making value of the measure (e.g. are you making better decisions as the result of knowing this) greater than the cost?

Reply

If we do not assign points to Defects , how do we quantify the failure work? Some of the team i worked with got lots of value by assigning points since it made the failure work visible to the team and stakeholders, while doing release planning we also get a feel of what is it going to take to clean the product from defects .

Reply
Agilemania

A common question that we get from teams who are new to agile is whether you should assign points (sizes) to defects. From a Disciplined Agile point of view we know that the answer will vary depending on the context of the situation, or in other words “it depends.

Reply
Charlie Coleman

I think confusion around all this simply traces back to the fundamental understanding by the team: what is a Defect and what is not. A Defect should be simply and directly defined as an Acceptance Criterion that was missed. If, at a demo, a user says “Oh, those values are too close together on-screen” or “I do not like the colors” – but neither of those were in the AC of the story, then what they have just described is a new story request. If there was an AC that stated “… progress should be based on the days of the month of sales” – but it turns out the code always used 30days in the calculation, that is indeed a Defect. This makes it clear that a Defect has direct business value (otherwise there would have been no need for the AC to begin with). Now, if the Defect was identified before iteration end, and there is time to work it, then you simply move the story status back from “Done” (or whatever) to In Progress and work it. No new points assigned in that case. But if the iteration ends with “Done” stories that have missed AC, then yep, write up a defect and put points on it. And clearly, any other type of additional request is simply another user story. Then what you do is you prioritize your Defects just like other User Stories. It is possible that a specific AC at the start of an iteration dropped dramatically in Business importance by the end of the iteration (or other stories rise quickly).

Taking this approach will yield several things:
– the whole team will pay very close attention to AC. And thus, AC will increase in quality throughout the project
– users will become aware of costs of their ‘little tweeks’ (Agile means flexibility, not free)
– developers will focus on the real needs (AC) vice cute ‘cool’ things to do in code
– the team knows EXACTLY what to expect. If an AC is missed: defect. If a tweak is desired: new story. And since everyone knows the focus is on the AC, there are rarely arguments about which is which.
– this can foster that true TEAM spirit: e.g. I’ve seen developers point out potential enhancements users did not think of (new story), and users critique each other’s AC statements to remove ambiguity. When that happens, the team starts firing on all cylinders, they really trust each other, the typical walls and blame games almost vanish.

Some might say that AC are often vague (or even they should be vague). But I have never seen that approach work. It leads to the worst Agile environments in my experience. It should be noted that a key premise of Agile is that you are using “seasoned” developers (10+ years of experience), and the base reasoning for this is over time, you get a “feel” for what users want when they say something, and the kind of User Interface that will work. Those seasoned developers also know when something sounds vague – sometimes intentionally vague, and needs to be cleared up. If the majority (I’d say 80%+) of the dev staff as “seasoned” you may well be better off going with waterfall. I saw one so-called Agile project where the dev team spent the first 2-4 days of the iteration creating detailed hourly estimates and task definitions for each user story, and then creating a MS Project schedule for the rest of the iteration. Yeah – all that time on non-producing tasks. They got into that habit because users kept identifying “defects” at the end of every iteration – but over 95% of them were not really defects. The young dev team simply did not know how to handle such a situation and the Scrum Master was not very strong (because of other team personalities). Once I was able to infuse the definition of a true Defect, things went much smoother (but we still lost all that time every iteration because dev staff were still shy of being beaten for supposedly not getting things done).

Inexperienced developers will be the ones putting out shoddy work, not because of their skills, but because they do not yet possess the ability to really link user statements with software function. And clearly, keeping Defect counts in plain sight is an powerful TEAM incentive: and the Scrum Master should make sure the WHOLE team (including users) take ownership when Defects occur (developer: what question should I have asked, user: how should I have stated that more clearly).

In the end, one of the most often repeated challenges I’ve seen in Agile attempts is users, and user group upper management simply assuming that Agile projects will cost less. And that drives them to the thinking that any “defect” that is found is “free” to fix. All that is horribly wrong of course, but it is difficult to break through the hype-bias those groups have built up. The best approach I’ve found so far is to simply keep track of all work: including Defects. By clearly defining what defects are, and putting points on them, you ‘normalize’ them into the project and, again, people start knowing what to expect. Where the strength of Agile comes in is when suddenly something in the business pops up and the users get frantic. They think that the project is hosed now because half their requirements (all of which are still on the backlog) are now meaningless. To which I simply say: hey, no problem, define the next thing you want us to work on and that other backlog stuff will just get pushed down. After that happens a couple times on a project, the right thinking in the user group starts to emerge.

So, to close, I’ll link this response to the above flow chart. Basically, using my definition of Defect, the flowchart would almost never be used. There is no subjective question of “is it substantial” – if it was an AC, it is substantial. Period. There is no question of “found by dev team” – if the iteration is not over it simply means they need to complete the AC. If the iteration is at the end, you better believe users will “find” it too (and the same applies for “found by test team”). The “found in production” is Non Sequitur: if users passed all the AC of any story, the story is truly finished/complete. If the production problem can clearly be shown to have violated an AC somewhere it is absolutely a Defect – it just means the dev team and user team failed in the validation process of the story. Regardless, it is a Defect and should be logged, sized, and prioritized. If the production problem cannot be traced to an AC, just write it up as a story and prioritize as usual.

Using the simple, clear definition of Defect and assigning points to “all work” removes the usual bickering and stops “self-protection processes” from developing. And that means a much higher percentage of time is spent producing activities.

Reply
Shreya Shinde

Good question and I liked your response. I think it is a frequent question where we discussed in our organization.

Reply

Leave a Reply to Charlie Coleman Cancel reply

Your email address will not be published. Required fields are marked *

Categories

Archives