Start Measuring Success With These 5 Agile KPIs-EN

As the saying goes, “You can’t improve what you don’t measure.” Metrics (operational), and more specifically Key Performance Indicators (KPIs) or Critical Success Factors (strategic), are about measuring progress toward your goal. Without them, you cannot know if your AS IS (current state) is making progress and whether you are on track to reach your TO BE (future state).

The thing that makes KPIs so elusive is that there are so many things that could be measured. KPIs can be strategic or operational. They can be based on output versus outcomes or process efficiency versus deliverables. They can be qualitative versus quantitative. They can focus on the leading versus the lagging. Or, they can measure the efficiency or the effectiveness.

Another thing that further complicates software development metrics is that the output is a widget and no two software outcomes are ever exactly the same (some may be similar but not the exact same). Even COTS projects have different configurations and are used differently in every environment.

Regardless of whether you are using Waterfall or Agile, many of these challenges can be addressed by learning about software development metrics. However, an Agile environment adds a new dimension that is not in your traditional menu of metrics. For more, read my Do Your Agile Project Metrics Measure Up? Why Traditional KPIs Won’t Work blog post.

As people go to great lengths to create agile environments, the need to quantify and measure success is a reasonable desire. Many organizations naturally think that measuring the following will determine their “agile” success:

• Velocity
• Team capacity
• Product owner’s time with the team
• Teams’ time in a co-located area
• Backlog growth
• Defect resolution rates
• Daily stand-up attendance
• Demos given
• Retrospectives held
• Testing occurring during the sprint
• Software delivered at every sprint

At a minimum, it probably is important that you are accountable for everything in this list. However, these measurements alone could stunt your agile growth because they focus on “doing agile” and not “being agile”. These metrics don’t communicate if you’re promoting customer collaboration, building trust, sharing knowledge, providing cross-functional training, prioritizing on high value, producing high quality software, fostering interaction, being responsive to change, and nurturing an engaging work environment.

For too long, metrics have put an emphasis on outcomes that are the result of unhealthy working environments, burn out, attrition, turnover, and work quality deterioration. Let’s look at some agile metrics that help measure and monitor more than ceremonies and focus on how the work is being done… metrics that measure and promote a healthy agile environment.

Being Agile KPIs

Metrics today need to encourage the type of behaviors that cultivate the “being agile” world many organizations say they are striving towards.

Behavior

Methods of Measurement

Individual Behaviors

Knowledge Sharing, Nurturing Others, Providing Opportunities for Team Members, Reaching Out, Challenges/Supports Others

360 Team Evaluations, Team Awards and Recognition, Individual Points Gamification

Team Behaviors

Respect, Trust, Honesty, Engagement, Determination, Collaboration, Passion, Initiative, Stretch, Transparency, Creativity/Innovation, Continuous Improvement, Swarm/Hovering Behavior

Niko-Niko Calendar, Caught in the Act Award, Team Points Gamification, Team Health Survey

Team to Team Engagement

Customer Collaboration (Face Time), Expectation Baseline, Interaction, Initiative, Responding to Change/Complexity, Outreach to Other Teams, Shared Services

Customer 360 Team Feedback, Shared Services 360 Team Feedback, Unplanned Request Metrics, Community of Practice, Guilds, Squads, Tribes, Dojo Activities

Outcomes

Growth, Innovation, Revenue Creation, Retention, Keeping the Lights On (KLO), Reduce Cost, MVP, ROI, Cost of Delay, Diminishing Returns

Post Implementation Evaluation, Business Value Burn Up, Feature Demos

Individual Behavior

Behaviors

Knowledge Sharing, Nurturing Others, Providing Opportunities for Team Members, Reaching Out, Challenges/Supports Others

Methods of Measurement

360 Team Evaluations, Team Awards and Recognition, Individual Points Gamification

Team Behavior

Behaviors

Respect, Trust, Honesty, Engagement, Determination, Collaboration, Passion, Initiative, Stretch, Transparency, Creativity/Innovation, Continuous Improvement, Swarm/Hovering Behavior

Methods of Measurement

Niko-Niko Calendar, Caught in the Act Award, Team Points Gamification, Team Health Survey

Team to Team Engagement

Behaviors

Customer Collaboration (Face Time), Expectation Baseline, Interaction, Initiative, Responding to Change/Complexity, Outreach to Other Teams, Shared Services

Methods of Measurement

Customer 360 Team Feedback, Shared Services 360 Team Feedback, Unplanned Request Metrics, Community of Practice, Guilds, Squads, Tribes, Dojo Activities

Outcomes

Behaviors

Growth, Innovation, Revenue Creation, Retention, Keeping the Lights On (KLO), Reduce Cost, MVP, ROI, Cost of Delay, Diminishing Returns

Methods of Measurement

Post Implementation Evaluation, Business Value Burn Up, Feature Demos

The Methods of Measurement in the last column are illustrations of the Gilb Measurability Principle: “anything you need to quantify can be measured in some way that is superior to NOT measuring it at all.” In other words, a measurement does not have to be perfect or even very precise, as long as your intent is to get a quantitative handle on something that was previously purely qualitative; the important thing is to take that first step toward quantifying.

Measuring Team Members – Individual Behaviors

Create team awards and recognition. (If this is something that already existed in your non-agile environment, make sure to revise the criteria for the rewards and recognition to agile values.)

I recommend creating team awards around those who take time out of their individual task to help a team mate. In a 360 Team Evaluation, ask team members (and only those who work together) if their coworkers are team contributors via a survey. The chart below can help identify team contributor behaviors for which to place a nomination… and behaviors that are not so desired.

Distraction

Individual Contributor

Team Contributor

Team Disruptor

Level of Control

Out of control

In control

Shares control

Dominates, dictator

Knowledge

Uninformed, out of the loop

Informed

Shares information

Withholds information

Support

Needy, helpless, takes but doesn’t give back

Self-reliant

Helpful, reaches out

Takes over or offers no help

Confidence

Anxious, uncertain

Secure, confident

Reinforces, installs confidence in others, coaches, mentors

Arrogant, unapproachable

Ability

Barely enough, Just enough

Capable

Trusts others’ capabilities

Pushy, conceited, “I’m right” attitude

Problem Solving

Fearful

Risk Taker

Offers others challenges, Supports efforts

Reckless, Self-absorbed, My way, Disregards how action impacts the team

As a group, the team determines how often the survey will be conducted, and the subsequent awards will be given.

As straight forward as that might seem, I have seen teams sabotage the nomination. In one environment that I worked in, a box was set out for team members to nominate each other. Several sprints went by and not one name was in the box. Many team members possessed ill feelings toward the transformation and longed for the days of being recognized as an individual contributor. Because no one was nominated, it was clear that coaching was needed to help them see the value in team contributor characteristics.

It’s been several months since the team received coaching, and now they regularly nominate members of their team. It is worth noting that the team did decide on a more formal, annual 360 review. Each team member gets to rank and vote on the other team members. No one outside the team votes. Everyone on the team then sees purely what their team members think of them and can adjust their behaviors accordingly.

Measuring Team Behaviors – Team Health

Agile benefits are realized when you have stable, high-performing team. To create a successful agile environment and culture, you need to measure the team’s health often and be ready to adjust when the numbers say the team is unhappy or unhealthy.

Something like team health or well-being might generally been seen as entirely subjective and thus impossible to measure and track. However, a technique called The Happiness Index, or the Niko-Niko Calendar, is a practice to help identify trends in the team’s attitude and tone.

It is also a great way to get the team engaged and talking about their challenges, not just general status updates. Sometimes the rich conversation derived from collecting this metric provides more value than anything else, especially if it leads to continuous improvement over time.

Measuring Team Behaviors – Team Engagement

The following chart is the foundation for a survey which was originally administered as a game using a platform called Kahoots®. Teams were quizzed on their understanding of why they were being asked to perform the various agile ceremonies.

Ceremony

Supporting Value/Principle

Product Backlog Grooming

Customer Collaboration – create a shared understanding of potential work

Sprint Planning

Working Software – gain consensus of the team’s capability and capacity

Daily Stand Up

Interaction and Individuals – open communication, daily commitments, evaluate priorities, synchronize the work

Sprints

Respond to Change – collaborate with the customer as a member of the team to jointly create a solution

Sprint Review

Customer Collaboration – showcase work, be accessible to the end users during the design process

Team Respective

Interaction and Individual – inspect the team dynamics, be self-aware, identify improvements

Measuring Team-to-Team Engagement

As one might predict, those who still had some resistance and hadn’t bought into the agile approach often answered the questions wrong. The purpose was not to convince or brainwash everyone into believing the purpose and intent of the ceremonies; it was to help the Scrum Masters make the ceremonies more meaningful.

This measurement is to test how well the team of teams are collaborating in an agile environment. There is a series of 5 questions for each category:

1. Are the sprint design changes being clearly communicated when they have an impact on others?
2. Is the Scrum Master someone that you can reach out to or that reaches out to you for clarification?
3. Is the staff open to cross-team collaboration within reason?
4. Is the work space conducive to cross-team collaboration?
5. Are the team skills as a whole in line with the work items assigned?

When coaching teams, I often point out that perception is important. If they disagree with the rankings, we talk about what they need to do to improve the problem area or improve the perception. Sometimes the team effort and contributions aren’t visible. This exercise may lead to a team-to-team discussion or may highlight roadblocks the Scrum Master needs to address.

Measuring Outcomes – Business Value

In Agile speak, the outcome is not defined by the product; rather, the product is the vehicle that leads to the outcome. There are a lot of non-agile ways to measure outcome, but a few of the most common are listed in the image below.

At the end of the day, if you can’t identify the outcome that benefits the business, then creating software for the sake of software is a waste of company money. Every member of the team should be fully versed and informed of how every item in every sprint contributes to an outcome that has business value.

The transparency element that is essential to the agile approach helps remove the disconnect between the contributions of each team member and the outcomes that the business finds valuable. To measure this type of metric, survey the team to be sure everyone has the same understanding. Some teams even use a point system to track and reward. Usually, there is a direct correlation between those who actively listen and ask questions.

Setting the Stage for Agile KPIs to Drive the Agile Mindset

My final cautionary note overall is that metrics can be used for good (to make the environment better) or for bad (to micro manage people). When a metric indicates negative trends, root cause analysis should be performed to understand the contributing factors. I strongly encourage a more organic approach toward agile metrics. You want to measure your success, not meter it based on a systematic pre-arranged list.

With creativity and openness to change, many of these agile metrics can work in your environment. Will all of them work? Perhaps not. But, try them. Stagger how you introduce them. Make sure people are being open-minded and give them a chance. Make sure everyone that is impacted understands the purpose and value of the metric. Ask for input upfront and feedback afterwards. One thing we do know is: if it doesn’t work, don’t waste more time doing the same thing. Learn fast and see what works in your environment.

Thank you,

Jacqueline