A Ramble About Metrics

You’re sitting in the director’s office with a nervous feeling.

“So, about those agile metrics I asked from you…”, she says. Your stomach churns and pulse quickens.

How will you show the progress of agile transformation? How will you measure improvement in teams and people? And how will you avoid the trap of shallow metrics like say:do ratios and velocity?

Whether you’ve worked as a manager, coach, or consultant, you’ve likely experienced conflict and confusion over “agile metrics.” Traditional metrics which emphasize personal productivity drive negative behaviors, encouraging us to stay busy with tasks over working together to achieve goals. Meanwhile, leaders feel dissatisfied with popular “agile metrics,” such as velocity and burn-down charts, when they fail to provide the insights desired.

This conflict between management and the information generated by metrics is further exacerbated when three common human desires for “measuring agile” inevitably occur:

• The need to prove “the new way of working” achieves more than “the old way of working.” This desire typically reveals itself through challenges like, “how will we know we’re more productive with agile?” or “how can you show me are teams are more efficient?”
• The desire for agile to deliver “more quality.” Often, we hold on to the premise that a new process (e.g., teams will work in sprints) is the key to unlocking quality. The scenario can be especially challenging in the absence of systemic knowledge about quality. “Our CEO is sending strong messages expecting quality to improve. Beyond defect counts and customers complaining, we’ve never had a good idea of what quality is. What do we do?” Sound familiar?
• The belief that process conformance is a good proxy for success. How we will respond when a senior leader challenges us with, “I want to know the maturity of teams and the adoption of process. How can you show me our agile transformation is successful?”

(Before we go any further thinking about these patterns, it’s worth asking an important question: why aren’t we measuring these outcomes before agile? Does it make sense to “measure agile” from a performance perspective if we don’t have the prior baseline to begin with? Anyways… back to the ramble.)

While there are many reasons these scenarios may be challenging–and frustrating–to solve, one possible contributor is the failure to distinguish between “success” and “improvement.”

By definition, “success metrics” are exactly that: the key measurement which reflects on the interaction (and collaboration) within the system to generate a successful outcome. By taking a measure-up approach to avoid focusing on individuals, we can see whether our system of work is delivering the result we need. Because we’re considering the success of the system, we likely only need one, possibly two, success metrics. Further, such metrics would be relatively long-term in persistence.

On the other hand, “improvement metrics” are useful until they no longer are. They exist to help guide us towards reaching a desired state, whether it be behavior, process, or performance. By definition, “improvement metrics” should be relatively short-term in lifespan and may change based on new information gathered. It’s easy to test for an improvement metric: we can state what decision(s) we will make as a result of the metric, plus what conditions would exist for the metric to no longer be useful.

An interesting thought which comes to mind: perhaps in the early stages of using agile (and assuming an absence of prior “success metrics”), our success metric might be the length of time before an improvement metric becomes obsolete.

I believe this structure of categorizing “success” and “improvement” metrics creates an environment where two beneficial things can happen:

1. We reduce the fear (i.e., gaming) of metrics through transparency of our intent to use metrics. Clear statements of intent, as well as conditions to make metrics “go away,” often create motivation to work with–rather than against–measurement.
2. The defined difference between “success” and “improvement” metrics can catalyze a helpful mindset change in people.

To illustrate the second point, consider a scenario which might be familiar:

A company shifts from a matrix system of work to teams working in sprints. Once this happens, a manager feels the need to measure team productivity and/or performance. After all, this is a new “process” and the manager wants to make sure it’s effective. To do so, the manager asks teams for a “say:do” ratio: the number of items/points/features/etc predicted versus the total actually delivered.

It’s likely the manager considers this measurement a “success metric.” Therefore, we ask her to explain how the metric represents the interaction of all the parts necessary for success. Assuming our goal is wildly successful software products which our customers love, we likely realize this metric ignores essential contributions of product management, customer support, the degree of disruption the team must manage, and other probable factors.

So, if not a success metric, might it be an improvement metric? We explore further:

• What decisions will we make with this metric?
  • If a “say:do” ratio is considered poor (the team doesn’t meet their forecast), what happens?
  • If the ratio is very good, what then?
  • Could decisions we make with this metric encourage people to “game” the metric or be less transparent?
• What conditions must exist to make this measurement no longer needed?
  • If a team meets their forecast 100% of the time, we likely no longer need it.
  • Is this condition an indicator we have achieved our goals?

I suspect these conversations might guide the manager to reveal her assumptions about measuring teams aren’t serving her ultimate needs. We then have the opportunity to explore together a more systemic need for measurement, most likely focusing on gathering information to improve, rather than perform.

In my experience with agile and “transformation”, whatever that might be sometimes,  I’ve found results tend to improve when we move away from “metrics to manage behavior” and begin using “metrics to help us understand behavior.” This framework of success and improvement metrics can help us achieve exactly that outcome.