Return to site

Measuring Developer Productivity

Transform your approach, and breathe a little easier

· leadership,metrics

One metric to rule them all?

I need you to give me one, simple metric that shows how productive our software teams are. - Your boss

This is my least favorite request in the entire universe. That is not an exaggeration.

The only way I could stop face-palming myself every time I heard this was understanding why it was being asked. Let's start there before we dig into how you keep this request at bay.

We'll never escape this question, and that's OK

Pro athletes make the sport they play appear very simple. However, everything changes when you go out to your driveway and try to drain that step-back three pointer like Steph Curry just did on TV. You shoot a couple more times and quickly realize there is a significant difference between you and Steph. You recognize how amazing it is that he makes those shots with two seven footers in his face. With this realization in hand, you focus on understanding how to do it yourself. What is you plan to shoot like Steph?

Maybe if you drink enough Gatorade, or buy his latest Under Armour shoes, you can start busting threes in your neighbor's face. There has to be a simple way to get that good. You just have to find that one thing to change so you can play like Steph or Lebron.

That's a silly idea, right? A sports drink and/or new shoes can't make you the best player in the universe. Is it sillier than trying to measure developer productivity with one metric? No Mars.

Becoming a world-class athlete is nuanced and requires thousands of hours of continuous training. There is no single thing you can do (I guess you could travel back in time and alter your DNA) to become great. Becoming world-class at anything, including building software, is the same way. If there was one magical thing you go to make teams great, no one would read articles like this. It's REALLY, REALLY hard. That's OK, especially once you understand the challenge in front of you.

Let's stay with the sports theme, and take a look at scouting/drafting pro athletes. Armies of scouts canvas the country looking for the next great draft pick. Yet with decades of experience evaluating talent and finding prospects with all of the right "measurables" (height, weight, speed, IQ, etc.), there are still draft misses. People try to market their own "one stop shop" for prospect ratings with things like star ratings (someone says a specific player is a 5 star recruit or a 3 star recruit). Check in on those 5 star recruits after a couple of seasons and you'll be amazed at how many aren't seeing any playing time.

Building great software is hard. Building teams that build great software products is even harder. The only thing harder than both of those is understanding the difficulty without ever doing it. When you don't have the experience of tackling something this difficult, you start looking for some indicators of success. What shoes can I buy my engineers that will maximize their productivity?!?

Single metric requests usually come from board members or executives that haven't had the opportunity to build products like you have. Once I understood all of this, I transformed from frustrated/confused to understanding/thoughtful. Your job becomes education. Explain the nuances. Walk people though the challenges. Help them understand why the concept of one, simple metric for developer productivity is a fallacy. Our instrumentation panel gives you everything you need to do that. Keep reading to find out how... :)

We are what we measure

Let's take the most extreme example: measuring lines of code as a determination of productivity.

If lines of code are how you decide to measure you development team, you will have the most verbose, complex system ever created. It will be a technological marvel. Then innovation grinds to a halt because of the complexity. You'll mostly likely blame the architect or senior engineers for allowing this to happen, but you planted the seed that grew this monstrous code base. You decided that lines of code were the thing that mattered most, so the team responded and gave you exactly what you asked for. Millions, or maybe billions, of lines of bloatware.

Focus on value

Do your customers pay you for lines of code? No, they pay you for delivering value equal to or greater than the money they are paying for your product(s). You must connect your development team with the value they are delivering. I labeled this "developing a commercial mindset" in a previous post.

To do this, we determine the common behavior(s) successful, happy customers exhibit. We then track those closely and aim to maximize that behavior with every story. Just as your core values define how your teams operate internally, use these behaviors to define what you work on and what order those items are in. Making this happen takes a lot of work from your product owners, but pays off massively. Product owners must spend time on more than just refining stories. They must understand the customers completely and then continuously educate your team on how your products help customers succeed.

The classic example is Netflix. The one behavior they focus on is maximizing the number of movies watched per customer. That's it. Everything they do centers on this. Not only does prioritization become simpler, but knowing that your prioritization was accurate gets a whole lot easier too. Imagine writing a story with an intent of increasing the speed of specific task by 5% and actually having a way to measure that. When you aren't trying to measure 5,000 different things, you can do this.

Early in my career I was exposed to this approach and it was fantastic. We were a small engineering team embedded in an insurance company. The founders of the company established 3 key behaviors they cared about: "how many policies, claims, and payments are we processing per employee?" Once we understood these success factors, that's how we measured our effectiveness and prioritized work. We started discussing things like "How many more policies can an underwriter process with this new change?" and "What are the results of that latest change? Did we get the 3% speed improvement in claim processing we were expecting?" We were then able to do the math and prove how much money we saved the company by increasing the performance of our application. Prioritization gets really easy when simple math tells you the ROI on every story. That's the commercial mindset you are looking for.

It's magic when you put all of this together. You have a team of commercially minded engineers focused on helping customers succeed in the ways that matter most. Customers are happy. Sales increase. Support calls decrease. People stop asking how to measure developer productivity. Let's chart a path to get there...

Build an instrumentation panel

We've all seen airplane cockpits. There isn't one giant dial that tells you if you are flying properly. A great pilot keeps an eye on dozens of gauges and dials all at once. These gauges & dials collectively provide her with everything she needs to understand if the plane is running smoothly. Each gauge is an indicator about one piece of the system's health.

Treat all of your metrics the same way a pilot does. They are a collection of indicators telling you the overall health of your system. When only one indicator is red, but the rest are green, there is likely nothing to worry about. That doesn't mean you ignore it completely. Just don't freak out. Spend time understanding what is pushing that indicator out of the normal zone. It could be that your team evolved and it's time to adjust your indicator's parameters or replace it with something else.

OK. Let's dig into a few key steps that help you build and maintain an instrumentation panel for your team(s).

  1. Make it public, then introduce it early and often.  The first thing a pilot does is acquaint themselves with the cockpit.  Where are all of the key gauges I need to pay attention to?  When a new employee starts, give them a walkthrough of your cockpit.  Tell them what all of the gauges are, and explain how they all work in unison.  This might be the first time the employee has seen something like this, so just showing them the gauges only confuses them.  Take time to talk through how they interact with each other.  Don't stop there.  Keep talking about it, with everybody, even your most senior people. Don't let any assumptions be made! 
  2. Keep evolving.  As you grow, new metrics might appear, or old metrics may become more important.  Don't assume that the metrics that matter for a 5 person team are still valid when your team is 50 or 500.  We just published a podcast episode covering what "just enough process" looks like.  Give it a listen for more details on how to know what is right for your team(s) as they evolve.
  3. Extend trust, repeatedly.  Seeing all of these metrics in a public setting will scare teams.  I have a few scars from not understanding this well enough.  Let's go back to the origin of this article.  Teams, over and over, get squeezed into a metric that determines how their productivity is perceived upstream.  This means that you likely have 50 different stories of engineers wrestling with this problem.  When you stand up and say "these are the things we are measuring", people will freak out.  It's up to you to not abuse these metics.  When your teams see you using these metrics as indicators of where you should dig in and help instead of overreacting to a dip in one number, everyone relaxes.  Like a good pilot, tap the gauge, check the other dials, and calmly react.  Freaking out helps no one, especially you!  
If you do those three things, consistently, you'll land in a fantastic spot. It takes continuous attention and education in all directions, so be prepared to spend a ton of time on this effort. Having been down this path, I can confirm that it is well worth the effort.

Wrapping Up

I'm passionate helping your team(s) succeed. Use the comments below to let us know how this article helped you. Take some time to dig into some of our Meta-Cast episodes centered on leadership. They will give you more insight into other key topics. If you are really excited to learn more, reach out to us and we'll help you build the team(s) of your dreams.

All Posts

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!