Scrum Teams Must Set Goals for Their Agile Metrics

Knowing what you want to find before you start looking seems like common sense. Yet many agile and scrum teams blindly gather metrics without really knowing what they want to learn from the data.

Erika | Target | Flickr

Erika | Target | Flickr

The way that most scrum teams approach agile metrics reminds me of the exchange between Alice and The Cat in Lewis Carroll’s classic Alice in Wonderland.

In this scene, Alice has come to a fork in the road and is unsure which way to go:

Alice: Would you tell me, please, which way I ought to go from here?
The Cat: That depends a good deal on where you want to get to
Alice: I don’t much care where.
The Cat: Then it doesn’t much matter which way you go.

Alice did not have a goal in mind while she was traveling along the road, and neither do most scrum teams when they set out to capture agile metrics.

Scrum teams can quickly become enamored with the amount of data that can be collected during a sprint. Story points, velocity, cost per story, cost per story point, throughput, number of stories per sprint, escaped defects, team health, and customer satisfaction are just a few examples off the top of my head.

Give the wide range and complexity of agile metrics that we could capture, we should accept a simple truth about these calculations:

The agile metrics that scrum teams collect are meaningless unless they are driven by a goal.

Why do you track velocity sprint after sprint? Because a training class said to? Or because you read about it in a blog post? No way!

There should be a problem that the scrum team is seeking to either solve or understand with every agile metric that they are collecting. With velocity, a scrum team could be trying to gauge their sustainable pace. Or perhaps they want to provide a means to help the product owner with release planning.

The actual goal is not as important as making sure that the team has one in the first place.

During a particularly difficult sprint retrospective meeting, one of my scrum teams noticed that they were struggling to deliver all of the stories that they committed to during consecutive sprints.

After some discussion, the team decided to track estimated story points per sprint and velocity. The difference between the two numbers would be called “found work”.

The scrum team decided that the goal of tracking these agile metrics – and particularly “found work” – was to measure changes in effort over the course of a sprint. The team felt that effort increased over time, which cause them to miss their sprint goals.

After a few sprints, the scrum team learned that the estimated effort was significantly increasing. But the numbers cannot tell you where the problems actually are. The root cause could have been one of many things:

  • Perhaps the scrum team was not properly grooming their product backlog items which led to “found work” during the sprint.
  • Maybe the scrum team’s definition of done was too stringent given their level of engineering practices which caused an increase in effort to deliver the stories included in the sprints.
  • It’s even possible that after digging deeper the scrum team learned that the relationship between the scrum product owner and the stakeholders was strained which led to poor communication and incomplete product stories.
  • Progressive elaboration could also have been taking place. The scrum team could simply be learning throughout the sprint and everything could be ok.

With the goal of their agile metric work realized, the scrum team focused on the problem during their next sprint retrospective. They were able to realize the high “found work” metric meant that too many epics were being added to the sprint backlog.

Not grooming their stories and slicing them in to smaller pieces hid much of the complexity that the scrum team was agreeing to take on sprint after sprint.

This particular team added a “3 touch rule” for stories to their definition of done. In order for a story to be eligible to be added to a sprint backlog, it had to be groomed and estimated at least 3 times by the team. This change led to the scrum team being more consistent at hitting their sprint goals.

Metrics are alluring, but they can also be deceptive and more importantly abused. Scrum teams should be intentional about how they collect agile metrics, the goal of the data, and what decisions will ultimately be made. Otherwise, why go to all the trouble of collecting the data in the first place?

Question: Does your scrum team collect metrics that do not have clear goals assigned to them? How are these agile metrics used? You can leave a comment by clicking here.

Scrum: Velocity Measures Story Points, Not Productivity

Scrum teams rely on metrics to inspect and adapt. Taken too far, the metrics tend to lose value and in some cases cause harm.

Robert Donovan | Velocity |

Robert Donovan | Velocity |

Last week I wrote about 5 agile metrics to measure high-performance scrum teams. At the top of the list was the most controversial metric that scrum teams use: velocity. This seemingly simple metric is just a measurement of how much effort a scrum team can complete within a given sprint.

Velocity is – in its simplest form – the sum of the story points completed in a sprint.

The temptation that many organization face is to turn velocity in to a performance metric.

For instance, I was recently asked what I thought about setting an objective to increase the team’s velocity by 10% by the end of the year. My answer: It’s a terrible idea.

  1. Encourages gaming:  An intelligent scrum team will realize that if they gradually inflate their estimates over the course of a year they will hit their 10% objective – WITHOUT delivering more value to the organization.
  2. Increasing velocity is not the goal of scrum:  We are interested in delivering valuable, high quality software to our companies in a sustainable way. How does a 10% increase in velocity get the scrum team there?
  3. Quality suffers as a result:  A scrum team that is forced to increase velocity will have to make tradeoffs to complete the additional stories required to get the numbers higher. Quality inevitably suffers when these types of decisions are made.
  4. Lots of unknowns:  Scrum team members leave the company, they get sick, and go on vacation. An estimation can also be way off – it is an ESTIMATION. If the scrum team learns something new about a story, velocity can dramatically go up or down.

Another idea that managers try is to compare the velocity of one scrum team with another. This is usually an attempt to normalize velocity across multiple scrum teams. It’s a meaningless comparison as scrum teams have different skills, goals, tools, and approaches to estimating work.

Flame wars often flare up over questions like: “Does a scrum team get credit for the story points related to bug during a sprint?”

Some scrum team members say, “Yes! We did the work and it should show on our velocity.” Others say, “No way, you already got credit for that story and are now cleaning it up. No double dipping!”

Mike Cohn did a great job of answering the question here. Essentially, he says pick whichever way makes sense to your team and be consistent. Easy, right?

Questions like these come up regularly and show a general lack of understanding of how velocity should be used.

Velocity is a limited metric that can only tell you what you achieved in the past and could possibly do in the future. During sprint planning it can help a team avoid over committing. During a sprint retrospective an up or down trend in velocity could lead to a discussion about the team’s definition of done or other team practices.

And there is the value that comes from tracking velocity. It is in the conversations that it causes the scrum team to have. These discussions are opportunities for the team to inspect and adapt their practices. The new insights will hopefully lead to more valuable, working software being delivered to the organization.

Which is important because according to the Agile Manifesto, “Working software is the primary measure of progress.”

Question: Which metrics do you use to drive conversations on your scrum team? You can leave a comment by clicking here.

5 Agile Metrics to Help Scrum Teams Improve

Scrum practices rely on empiricism. We inspect our work and adapt our practices to achieve continuous improvement. But what should a scrum team use in order to improve?

Jorge Franganillo | Calculator |

Jorge Franganillo | Calculator |

I enjoy Bob Galen’s blog quite a bit. In a recent post – “2 Dozen Wild & Crazy Agile Metrics Ideas” – he presents 24 wild and crazy agile metrics that a scrum team could add to their agile toolkit. It’s an excellent read that made me pause and wonder what scrum teams really should be measuring.

After some careful thought, I’ve landed on what I think are the 5 agile metrics that teams should focus on and the benefits that these metrics can bring to scrum teams:

  1. Velocity: When we track velocity, we are expressing how much effort (story points) a scrum team can get to “done” during a sprint. With velocity, the trend is more important than any individual measurement. As the trend stabilizes, teams can forecast their product backlog. Release planning also becomes easier for the product owner.
  2. Defect Density: The number of bugs discovered during a sprint. This measurement is a call back to the scrum team’s commitment to quality. The lower the number of bugs the better. Again, the trend is important. An increasing number of bugs sprint-over-sprint could mean that the team is taking on too much work. A downward trend could point to changes in the definition of done improving quality.
  3. Customer Satisfaction: How happy is your customer? The simplest method I’ve used is a simple web application that asked customers how they feel about the current sprint. They could select a smiley face, a frown face, or a neutral face. If a customer picked a frown face they were asked to provide additional comments. The goal is measure satisfaction over time and to also address negative feedback quickly.
  4. Team Satisfaction: Is your team happy? This is a measurement that could be gathered at the end of each sprint retrospective. The measurement could be the results of a “fist of five” question, or it could be a survey similar to the customer satisfaction metric. Scrum teams should keep an eye on this trend and use the “5 Why’s” and other techniques to get to the root cause of whichever way the trend is going.
  5. Value Per Sprint: This metric measures how much value the scrum team is delivering back to the customer/business/organization. One of the components of a story card is value. Scrum teams can use this value and measure a trend. The product owner could also provide a value in dollars that represents the impact of the sprint. A downward trend in this metric could indicate that lower value features are being implemented and that it is time to stop development on the product.

Keep in mind that these are team metrics, not individual ones. I’ve also put a much higher value on trends than on individual data points. Finally, agile metrics and trends should be taken with a boulder of salt. You won’t find a silver bullet buried in the data that solves all of the scrum team’s problem. But hopefully you’ll use some of these agile metrics to kick off an interesting conversation with the team.

**UPDATE:  These metrics are for the team to use to enable continuous improvement. They are not for managers to use to drive artificial “improvements” within the team. In future posts I’ll discuss my views on metrics belonging to the team, not management. Thanks to Charles Bradley for his thoughts and messages on this important distinction.

Question: Well those are my top 5 agile metrics. Have I missed your favorite? Is there a must have measurement that you would add? You can leave a comment by clicking here.