Category Archives: TDD

The Stability Index: Focusing on Release Stabilization

While recently working with Juni Mukherjee on a team that is focused on finding ways to extending and increasing the value of a large legacy platform she brought up what I thought was a brilliant idea. We had been working on creating metrics that have tension with each other to drive continuous integration effectiveness from the component level up to the deployed system and continue to generate value to users. Juni’s research and the team discussions on the topic went through multiple scenarios with different metrics to better understand how they balance and/or mislead. As we all know any metric can be gamed. And not only that, but a metric is not always valuable in a context even though it is extremely valuable in another. During the conversation Juni blurted out that what she was looking to come out with was a “Stability Index”. This brilliant phrase along with the outcomes of our team discussions lead me to think that this is a valuable way to look at quality measurements alongside other release constraints to support delivery of continuous value.

This article is a first attempt at putting The Stability Index down on paper as it is already in use, to some degree, in Juni’s organization. In the past, organizations and teams I have worked with have come up with similar approaches that allow us to balance effective quality signals. This should lead to early detection of what are usually weak signals of quality issues that many times are found too late or whose symptoms are seen away from the signals origin.

Goals of The Stability Index

Stability Index is a function of signals that are observed in pre-production and production environments. The purpose of calculating a Stability Index is two-fold:

  • The Stability Index is an indicator of how an organization is progressing towards its business goals. For example, if Stability Index goes up, Cycle Times reduce, Release Stabilization Period goes down and Customer Retention improves.
  • The Stability Index also reveals any correlation of pre-production signals to signals observed in production. For example, when Code Coverage goes up, the number of defects found in production goes down.

If proper balance is given to these pre-production and in production signals than this should lead to a stable application platform that continuous to deliver value at a sustainable rate as new business needs arise. Of course, this does not mean that business needs will be continuous or steady so there is still a potential to impact The Stability Index but by keeping balance, a team or teams should be able to manage fluctuations in business needs more effectively.

Ultimately, creating an implementation of The Stability Index is deciding on metrics that produce effective pre-production and in production early warning signals. The following sections will go into detail about the metrics which were are initially used in this implementation of The Stability Index.

Pre-Production Signals

Pre-production signals are focused towards technical craftsmanship of engineering teams.

Percentage of Broken Builds

This metric is used to objectively measure behavior patterns of engineers who may not be using effective gating criteria before checking in code. Build breakages could be while building source code, compiling, unit testing, distributing artifacts, deploying images or further downstream while testing for functional correctness, integration issues and performance gaps. Irrespective of where the breakage is, code will not be able to flow to production through a fully automated pipeline unless engineers test code changes in their local sandbox environments before checking into the common repository.

 Code Duplication

This metric is an early indicator of software debt that indicates that software programs have significant sequences of source code that repeat and this calls for refactoring. This has a high business impact in terms of maintainability.

Cyclomatic Complexity

This is a software metric that indicates the conditional complexity of a program (function, method, class etc.) by measuring the number of linearly independent paths that can be executed. This is also an early indicator of software debt and is very expensive to the company when it comes to getting new employees up to speed.

Code Coverage

Code Coverage can be either Line/Statement/Decision/Branch/Condition coverage and is a measure of how effective test suites are in certifying software programs. Code Coverage also indicates that 100% Line or Statement Coverage may give a false sense of security since all lines may have been covered by tests although important decisions, branches and conditions may not have been tested.

The caveat here is that a high code coverage percentage does not guarantee bug-free applications since the primary objective of tests should be to meet customer requirements and all customer use cases may not be exercised by tests even when Code Coverage is at 100%.

Test Cycle Time

Test Cycle Time is a measure of the time it takes to execute the entire test suite before either reporting bugs that would cause another iteration or software can be certified for production. For the same number of tests, the Cycle Time could be high unless tests are designed to be independent of each other and are launched in parallel. As an aside, for tests to be launched in parallel, the test environments can often become a bottleneck in terms of providing the required capacity and reliability. Virtual environments may not offer a guaranteed share of the CPU whereas shared clusters in distributed environments may queue jobs and hence make the execution time long and unpredictable.

Production Signals

Production signals are focused towards the customer.

Customer Delight / Satisfaction

It’s all about the customer. Software is released to meet the customer’s needs and to leave the customer delighted and craving for more. Although “Delight” can be subjective at times, surveys are an effective way to measure satisfaction. An example of customer dissatisfaction could be that albeit the software behaves correctly, it takes more number of clicks to perform the same activity than it used to take in the previous version. Defects reported by the customer are a good measure of this metric.

Defect Containment

This is an important trend to watch out for since customers can be inconvenienced if their support tickets are queued up. Moreover, defects reported in production that translate into code and configuration errors should be fixed by the engineering team within acceptable SLAs depending on the severity of the issues. Being able to iterate fast is one of the key factors for customer retention.


Systems could go down due to hardware incidents like router malfunction and disk crashes or due to software inadequacies like fault tolerance not being built in. Either way, downtimes cause revenue losses and are a critical contributor towards Stability Index.

Relationship of Signals to The Stability Index

Each of the above trends has a bearing on the Stability Index. Some trends are directly proportional to the Stability Index (), while the trends of others are inversely proportional (). For example, when Code Coverage goes up, it has a positive impact on Stability Index. On the contrary, when Code Duplication goes up, Stability Index goes down.

The relationships of all the metrics with Stability Index are illustrated below.

Pre-production Metric Relationship to Stability Index
Percentage of Broken Builds  
Code Duplication  
Cyclomatic Complexity  
Code Coverage  
Test Cycle Time  
Production Metric Relationship to Stability Index
Customer Delight / Satisfaction  
Defect Containment  

Recognizing Software Debt Talk at Beyond Agile Meeting

A couple days ago I spoke at the Beyond Agile group meeting on the topic of “Recognizing Software Debt”. Early in the presentation we ran an exercise to get a feel for the effects of software debt that was original created by my friend, Masa Maeda. Here is a link to the exercise:

The exercise went great even though I was using the audience as guinea pigs to run the exercise for my first time. Below is the slide deck that I used as a backdrop for the presentation.

Delving into Technical Debt – Cutter Article

The following is an except from the article authored by Israel Gat and myself named “Delving into Technical Debt”:

Many of the findings and the recommendations we make in Cutter technical debt engagements are broadly applicable in concept, if not in detail. There is commonality in the nature of the hot spots we typically find, the mal-practices we identify as the root causes and the ways we go about reducing the “heat.” Granted, your technical debt reduction strategy might dictate investing in automated unit testing prior to reducing complexity, while your competitor might be able to address complexity without additional investment in unit testing. However, the considerations you and your competitor will go through in devising your technical debt reduction strategies are fairly similar.

It is this similarity that we try to capture in this Executive Update. Some of specifics we recount here might not be applicable to your environment. However, we trust the overall characterization we provide will give you, your colleagues and your superiors a fairly good “3D” picture of how the technical debt initiative will look like in the context of your own business imperatives and predicaments.

As a Senior Cutter Consultant, it has been a pleasure working with Cutter to release this Executive Update from which the excerpt cited above is taken. For a free donwnload, click here and use the promotion code DELVING. Let us know what you think about the article in the comments section of this post.

Treating Software as an Asset

William Caputo wrote a passionate blog entry on why “Software is not an asset” here:

Although I entirely agree with ideas discussed about refactoring and removal of code, I do not think that the blog entry substantiates his claim that software is not an asset. An asset is:

Anything tangible or intangible that is capable of being owned or controlled to produce value and that is held to have positive economic value is considered an asset

Much of the software out there has a positive economic value either in productivity, revenue generation, cost reduction, or other types of economic value. I see removal of code and refactoring code into a more desirable design in order to support changing business needs as essential to countering depreciation of software assets. This was the foundation of Managing Software Debt: Building for Inevitable Change.

When we describe software only as a liability then we also provide reasoning to business that those who work on the software are part of a cost center. This is usually not the best way to operate any software development organization. The focus should be on optimizing the value that existing and newly created software assets support. Rather than only a liability, we can think of assets as what they are defined to be:

Assets are equal to “equity” plus “liabilities.”

The equity can still be there but if the liability is becoming so large that it outweighs the utility of the equity then we have a problem. We should counteract this decay in value by Managing Software Debt effectively as we produce new communications between the software and hardware assets we deploy.

Towards a Push-Button Release

This presentation is being delivered as a 45-minute lecture and discussion for a company-wide tech talk today. It contains 2 case studies that revolve around moving to a push-button release that reduced the whole product company’s release cycles from 6 months to every week and the effects of a “No Defect” policy on a team’s productivity.

Managing Software Debt in Practice Presentation

Today at the Scrum Gathering in Seattle, I held a session on “Managing Software Debt in Practice” where we got into:

The presentation had too much for the less than 90 minutes that we had for the session. I did not get into scaling Scrum team patterns and heuristics to manage software debt at scale and also less around testing than I’d hoped. Hopefully it was useful for the participants and they got at least one new idea leaving the session. It is difficult to take a 1-day workshop and create a less than 90 minute talk, as I learn again.

Interview: Assessing ROI of Addressing Software Debt

This week, an interview from with me came out on the book Managing Software Debt: Building for Inevitable Change. The interview has 2 parts. The first is a discussion of software debt and the second focuses on addressing software debt. Here are links to the interview:

Impediment Management and the Agile Triangle

The Agile Triangle, a concept that was discussed by Jim Highsmith in his book Agile Project Management: Creating Innovative Products, describes how teams and organizations can go beyond the traditional Iron Triangle in defining project success criteria. In the Agile Triangle, the traditional project constraints of schedule, cost, and scope are confined to one point of the triangle. Balancing out the triangle are quality and value. Agile software development practices have provided huge innovations in delivery of working software from an internal and external perspective. Practices such as automated unit testing, Test-Driven Design (TDD), and Continuous Integration (CI) have transformed our industry and enabled new go-to-market strategies for the business of software products.

Agile Triange (Jim Highsmith*) adding Software Debt
Agile Triange (Jim Highsmith*) adding Software Debt

When looking at the Quality point on the triangle, this is where I see the realm of Managing Software Debt providing a strategy to support greater value generation within the defined project constraints, as shown in the diagram above. Creating a software debt management strategy involves thinking about feedback mechanisms for each type of software debt: technical, quality, configuration management, design, and platform experience. For technical debt, which tends to be focused on code, we can use tools such as Sonar to provide feedback on changes. For quality debt, we can look at defect containment metrics, bugs escaping iterations and into releases, as a feedback mechanism to trend. Configuration management debt could include the ability for an application to use 2 scripts, deploy and rollback, for all environment deployments and promotions along with monitoring branching in source control for potential integration issues (Sonar is integrating this capability, as well). Design debt can be monitored using Sonar and design assertion frameworks such as TisUgly for Java that I created and never marketed well a couple years ago. Finally, we can monitor the effectiveness of our platform experience debt management strategy based on impediment resolution cycle time or other team and management measurements.

This last point about impediment management is incredibly important and doesn’t get enough attention in the Agile community, in my opinion. In fact, as someone who is experienced in supporting enterprise Agile adoptions for defined business objectives, impediment management can be a critical element of sustained Agile software development effectiveness. When organizations hit a wall with impediments, mostly organizational impediments to delivering increased value to customers, they become susceptible to reverting towards past practices to reduce noise and ignore the effects of these impediments. The diagram below shows how impediment management is a wrapper around the Agile Triangle to support increased and sustained value delivery over time.

Agile Triange (Jim Highsmith*) adding Impediment Management
Agile Triange (Jim Highsmith*) adding Impediment Management

Impediment management comes down to communication. How are impediments raised? When impediments are not within the team’s realm of control to resolve, how does management support their resolution? At enterprise scale, is there a need to provide visibility to more than 1 or 2 levels higher in the organization? If so, how do these enterprise impediments get handled at strategic levels in the organization? What is your impediment management strategy?

Please let us know your thoughts on the relationship of impediment management and Agile Triangle. Also, tell us in the comments section how you see impediments managed in your organization. What is going well? What could be improved? How are they escalated? We look forward to hearing your thoughts and experiences.

Training Schedule 2011

Happy New Year! We are looking forward to a great 2011. We have a great lineup of training planned, including our CSM and CSPO offerings, AgileEVM training, and some new trainings.

New Training Offerings

Along with our current training offerings around Certified ScrumMaster (CSM) and Certified Scrum Product Owner (CSPO) courses, we will also be offering training for:

  • Agile Portfolio and Release Management using AgileEVM
  • Managing Software Debt Workshop (based on our book)
  • Online training and micro-consulting

Although we have been training these unofficially for some time now, these are new official offerings, we are looking for companies and individuals who are interested in taking these courses for a discount. Please let us know if you are interested in test-driving any of these offerings.

CSM and CSPO Training

Lets not forget our CSM and CSPO course listings. You can find a full list of our classes on the Scrum Alliance and on our web site. We will be holding CSM and CSPO classes back-to-back in San Francisco on the following dates:

  • February 15-18
  • March 22-25
  • April 26-29
  • June 14-17
  • August 23-26

Of course, our CSPO course will include introductory training on the use of Innovation Games® to gather data from your customers and stakeholders. We feel strongly that a healthy relationship with your user community and stakeholders is essential to delivering great products.

AgileEVM Training

In fact, we have been working with our customers over the past 6 months on enhancing and improving our own product, AgileEVM. You can take a tour of AgileEVM on our training video page. Please register for a free account today if you don’t already have one. You can manage up to 2 active releases with your free account so please try it out and let us know what you think at AgileEVM Support.

We are looking forward to a great year in 2011. Please let us know if you are interested in our training or consulting services at Sterling Barton.