Do Your Agile Project Metrics Measure Up? Why Traditional KPIs Won’t Work-EN
Jacqueline Sanders-Blackman
- Business Agility
- Article
Organizations who are embracing scaled agile and digital agility need to recognize something very important (if they haven’t already): in order to be successful, a paradigm shift must occur in every aspect of corporate culture. In fact, the concept of business agility encompasses:
- Portfolio/Program/Product Management
- Development/Deployment
- Budgeting
- Talent Management
- Project Management
These areas all have to be re-engineered, re-modeled, or re-purposed. Establishing a long-term agile culture cannot be realized by maintaining and thinking of metrics “the way you have always done them”. Your metrics must also be re-engineered, re-modeled, and re-purposed. If not, how you define and measure success won’t achieve full business agility.
Metrics of the Past Have Taught Us A Lesson
Many traditional KPIs can instigate and proliferate bad behaviors in software development. In the past, programmers have measured their progress by how many lines of code they’ve written, quality was measured by the number of bugs fixed, and user needs were indicated as having been met by checking off a list of requirements.
The software development environment, which is legendary and infamously known as the Waterfall method, included several common KPI ingredients: performance, process, and delivery.
- Performance goals » fast delivery
- Process goals » lean and at the lowest cost
- Product or service delivery goals » all or nothing requirements, error free and perfect
On the surface these all seem relevant and worthy of measuring. What could possibly go wrong?
Goal that Drives Bad Behaviors | Unintentional Outcome |
|---|---|
| Deliver things fast | Deliver the wrong things fast; Sacrifice quality |
| Focus on low cost | Cut corners; Amass technical debt; Sacrifice quality |
| Make processes lean | Cut necessary and unnecessary resources and steps without differentiating them |
| Evaluating that the process and/or the product is “error” free | Expect perfect process and punish errors; Expect perfect products and discredit anything that is less than perfect (people will hide the errors) |
| Only accept all or nothing | Focus on checking the box and not prioritizing what was of highest value |
Metrics in themselves can result in people figuring out ways to play the “number game”. They will do anything to make the “numbers look good”. Unfortunately, bad metrics have found their way into the agile environment. It’s important to recognize and repackage them into something productive. Metrics today need to create the type of behaviors that cultivate the “being agile” world many of us are striving towards.
Hard vs. Soft Metrics
The metrics of the past focused on what is known as hard metrics. The metrics that support the agile definition of success require soft metrics.
Hard metrics rely on verifiable data points. They use quantitative data points to evaluate a typical work routine and how well the employees carry out their assigned tasks.
Unlike hard metrics, soft metrics use subjective data and interactive responses to determine effectiveness. Soft metrics stress the impact that human capital has on business outcomes.
Hard metrics aren’t all bad, but soft metrics create a necessary balance. Here are three hard and soft metrics that balance each other and can set a very different tone for the working environment.
Hard Metrics | Soft Metrics |
|---|---|
| Performance Goals – Fast Delivery | Healthy Teams |
| Process Goals – Lower Cost | Minimum, High Value Products |
| Product/Service Goals – Error Free, Perfection | Learn Fast and Adjust |
Another way to think about it is: hard metrics indicate if you are doing agile, soft metrics measure if you are being agile.
Agile Redefines Success
The Agile approach fundamentally redefined success factors for software delivery. People no longer work in silos and their success can no longer be measured in an individual capacity. If the product fails, the team fails, and everyone on that team fails. With the measurement of success based on a team’s effort, the ability to finger point is significantly reduced.
Let’s do a quick exercise to determine how you are measuring success. Make a check beside each criteria that you are using. Do you have more agile or legacy checks?
Agile Definition of Success | Legacy Definition of Success |
||
|---|---|---|---|
| Measure how well value was delivered | Measure how well the plan was followed | ||
| Measure product success | Measure project success | ||
| Measure if an outcome was achieved | Measure if the documented requirements were followed | ||
| Measure if quick feedback was received and adjustments were made quickly | Measure if changes were avoided and the solution was sold AS IS | ||
| Measure collaboration | Measure individual contributors to the team and target bottlenecks | ||
| Measure shared learning achievements | Measure individualized learning achievements | ||
| Measure and reward team cohesiveness | Measure and award individual contributors | ||
| Measure the Return on Investment | Measure cutting cost | ||
Agile project metrics are different because they are people-centric; they recognize that people drive the outcomes. It’s more than delivering a product in the agile mindset. It becomes what the team did (or didn’t do) in their pursuit of creating a product as a team.
Consider the Values Defined in the Agile Manifesto
Individuals and Interactions Over Processes and Tools
By just counting lines of code, developers don’t need to collaborate with their colleagues or even talk to others in the development life cycle as long as they are producing code – no matter if it’s even good, bad, or usable.
The intent of an agile approach is for developers to collaborate with their colleagues, talk through designs, and transfer knowledge as they are producing code. These actions will likely slow them down at first, but it is ok. I’ve seen people go from counting lines of code to now finding ways to make their story points good. In the long run, this team collaboration will only enable faster (more complete) work.
Working Software Over Comprehensive Documentation
Using bug fixes as measurement misses the point of why there are so many mistakes in the first place. In the Agile mindset, instead of counting how many defects you close and how fast you close them, count the days of being “error free”.
I’ve seen signs in factories that display the number of days without an accident. We should create similar radiators to measurement software quality. If a team can go 90, 60, or even 30 days with no reported urgent bugs, that’s a metric worth celebrating! If 30 days is not attainable, get to the root of what’s causing the urgent bugs in the first place.
Customer Collaboration Over Contract Negotiation
Don’t think that just because you have a lot of meetings scheduled that there is true collaboration taking place. In many environments people walk out of a meeting in agreement but once the work starts and something doesn’t go as planned (or something unplanned gets identified), the finger pointing starts. More time is spent trying to recall what was or was not decided in the previous meeting rather than trying to move forward to find a solution.
I’ve also seen co-located teams sitting side-by-side, but all wearing head phones. They can go the whole day without talking. Co-location does not necessarily equal collaboration.
A good way to measure collaboration is to consider how much time is spent paired with a team mate, how much time is spent working on the same work item, voluntary participation in group problem solving, and transparency of what you are working on: “working out loud”.
Responding to Change Over Following a Plan
Unfortunately, I’ve seen some teams have success measured by the product owner getting their wish list fulfilled. Usually, the teams fighting this kind of structure can’t express their opinions and never get the chance to say ‘no’. In these cases, the product owner is (of course) 100% happy because they are getting their way without ever being questioned. The plan they laid out was followed… project complete, right?
Responding to a change does not mean the team has no input or power to push back, especially when it’s justifiable. You want to use measurements that show the team is responding to change but that it’s not haphazard or reckless. Capacity, risk, and impact on the team all need to be considered.
What’s Next
At this point, if you are at least questioning or rethinking whether the metrics you are using are undermining the mental shift that is required in an agile environment, then you’ve accomplished the first step: admitting you might need to make some changes.
Design a framework to architect and measure success for your agile organization in our Agile Metrics and Value Management course.
The benefit of Agile is being able to learn quickly from new information and make the necessary adjustments as soon as possible. So, how do you measure success in an agile environment? Read my Start Measuring Success With These 5 Agile KPIs blog post!
Thank you,
Jacqueline
