Category Archives: Iterative Design

The Stability Index: Focusing on Release Stabilization

While recently working with Juni Mukherjee on a team that is focused on finding ways to extending and increasing the value of a large legacy platform she brought up what I thought was a brilliant idea. We had been working on creating metrics that have tension with each other to drive continuous integration effectiveness from the component level up to the deployed system and continue to generate value to users. Juni’s research and the team discussions on the topic went through multiple scenarios with different metrics to better understand how they balance and/or mislead. As we all know any metric can be gamed. And not only that, but a metric is not always valuable in a context even though it is extremely valuable in another. During the conversation Juni blurted out that what she was looking to come out with was a “Stability Index”. This brilliant phrase along with the outcomes of our team discussions lead me to think that this is a valuable way to look at quality measurements alongside other release constraints to support delivery of continuous value.

This article is a first attempt at putting The Stability Index down on paper as it is already in use, to some degree, in Juni’s organization. In the past, organizations and teams I have worked with have come up with similar approaches that allow us to balance effective quality signals. This should lead to early detection of what are usually weak signals of quality issues that many times are found too late or whose symptoms are seen away from the signals origin.

Goals of The Stability Index

Stability Index is a function of signals that are observed in pre-production and production environments. The purpose of calculating a Stability Index is two-fold:

  • The Stability Index is an indicator of how an organization is progressing towards its business goals. For example, if Stability Index goes up, Cycle Times reduce, Release Stabilization Period goes down and Customer Retention improves.
  • The Stability Index also reveals any correlation of pre-production signals to signals observed in production. For example, when Code Coverage goes up, the number of defects found in production goes down.

If proper balance is given to these pre-production and in production signals than this should lead to a stable application platform that continuous to deliver value at a sustainable rate as new business needs arise. Of course, this does not mean that business needs will be continuous or steady so there is still a potential to impact The Stability Index but by keeping balance, a team or teams should be able to manage fluctuations in business needs more effectively.

Ultimately, creating an implementation of The Stability Index is deciding on metrics that produce effective pre-production and in production early warning signals. The following sections will go into detail about the metrics which were are initially used in this implementation of The Stability Index.

Pre-Production Signals

Pre-production signals are focused towards technical craftsmanship of engineering teams.

Percentage of Broken Builds

This metric is used to objectively measure behavior patterns of engineers who may not be using effective gating criteria before checking in code. Build breakages could be while building source code, compiling, unit testing, distributing artifacts, deploying images or further downstream while testing for functional correctness, integration issues and performance gaps. Irrespective of where the breakage is, code will not be able to flow to production through a fully automated pipeline unless engineers test code changes in their local sandbox environments before checking into the common repository.

 Code Duplication

This metric is an early indicator of software debt that indicates that software programs have significant sequences of source code that repeat and this calls for refactoring. This has a high business impact in terms of maintainability.

Cyclomatic Complexity

This is a software metric that indicates the conditional complexity of a program (function, method, class etc.) by measuring the number of linearly independent paths that can be executed. This is also an early indicator of software debt and is very expensive to the company when it comes to getting new employees up to speed.

Code Coverage

Code Coverage can be either Line/Statement/Decision/Branch/Condition coverage and is a measure of how effective test suites are in certifying software programs. Code Coverage also indicates that 100% Line or Statement Coverage may give a false sense of security since all lines may have been covered by tests although important decisions, branches and conditions may not have been tested.

The caveat here is that a high code coverage percentage does not guarantee bug-free applications since the primary objective of tests should be to meet customer requirements and all customer use cases may not be exercised by tests even when Code Coverage is at 100%.

Test Cycle Time

Test Cycle Time is a measure of the time it takes to execute the entire test suite before either reporting bugs that would cause another iteration or software can be certified for production. For the same number of tests, the Cycle Time could be high unless tests are designed to be independent of each other and are launched in parallel. As an aside, for tests to be launched in parallel, the test environments can often become a bottleneck in terms of providing the required capacity and reliability. Virtual environments may not offer a guaranteed share of the CPU whereas shared clusters in distributed environments may queue jobs and hence make the execution time long and unpredictable.

Production Signals

Production signals are focused towards the customer.

Customer Delight / Satisfaction

It’s all about the customer. Software is released to meet the customer’s needs and to leave the customer delighted and craving for more. Although “Delight” can be subjective at times, surveys are an effective way to measure satisfaction. An example of customer dissatisfaction could be that albeit the software behaves correctly, it takes more number of clicks to perform the same activity than it used to take in the previous version. Defects reported by the customer are a good measure of this metric.

Defect Containment

This is an important trend to watch out for since customers can be inconvenienced if their support tickets are queued up. Moreover, defects reported in production that translate into code and configuration errors should be fixed by the engineering team within acceptable SLAs depending on the severity of the issues. Being able to iterate fast is one of the key factors for customer retention.

Uptime

Systems could go down due to hardware incidents like router malfunction and disk crashes or due to software inadequacies like fault tolerance not being built in. Either way, downtimes cause revenue losses and are a critical contributor towards Stability Index.

Relationship of Signals to The Stability Index

Each of the above trends has a bearing on the Stability Index. Some trends are directly proportional to the Stability Index (), while the trends of others are inversely proportional (). For example, when Code Coverage goes up, it has a positive impact on Stability Index. On the contrary, when Code Duplication goes up, Stability Index goes down.

The relationships of all the metrics with Stability Index are illustrated below.

Pre-production Metric Relationship to Stability Index
Percentage of Broken Builds  
Code Duplication  
Cyclomatic Complexity  
Code Coverage  
Test Cycle Time  
Production Metric Relationship to Stability Index
Customer Delight / Satisfaction  
Defect Containment  
Uptime

Recognizing Software Debt Talk at Beyond Agile Meeting

A couple days ago I spoke at the Beyond Agile group meeting on the topic of “Recognizing Software Debt”. Early in the presentation we ran an exercise to get a feel for the effects of software debt that was original created by my friend, Masa Maeda. Here is a link to the exercise:

http://www.agilistapm.com/understand-technical-debt-by-playing-a-game/

The exercise went great even though I was using the audience as guinea pigs to run the exercise for my first time. Below is the slide deck that I used as a backdrop for the presentation.

Delving into Technical Debt – Cutter Article

The following is an except from the article authored by Israel Gat and myself named “Delving into Technical Debt”:

Many of the findings and the recommendations we make in Cutter technical debt engagements are broadly applicable in concept, if not in detail. There is commonality in the nature of the hot spots we typically find, the mal-practices we identify as the root causes and the ways we go about reducing the “heat.” Granted, your technical debt reduction strategy might dictate investing in automated unit testing prior to reducing complexity, while your competitor might be able to address complexity without additional investment in unit testing. However, the considerations you and your competitor will go through in devising your technical debt reduction strategies are fairly similar.

It is this similarity that we try to capture in this Executive Update. Some of specifics we recount here might not be applicable to your environment. However, we trust the overall characterization we provide will give you, your colleagues and your superiors a fairly good “3D” picture of how the technical debt initiative will look like in the context of your own business imperatives and predicaments.

As a Senior Cutter Consultant, it has been a pleasure working with Cutter to release this Executive Update from which the excerpt cited above is taken. For a free donwnload, click here and use the promotion code DELVING. Let us know what you think about the article in the comments section of this post.

Treating Software as an Asset

William Caputo wrote a passionate blog entry on why “Software is not an asset” here:

http://www.williamcaputo.com/archives/000310.html

Although I entirely agree with ideas discussed about refactoring and removal of code, I do not think that the blog entry substantiates his claim that software is not an asset. An asset is:

Anything tangible or intangible that is capable of being owned or controlled to produce value and that is held to have positive economic value is considered an asset

Much of the software out there has a positive economic value either in productivity, revenue generation, cost reduction, or other types of economic value. I see removal of code and refactoring code into a more desirable design in order to support changing business needs as essential to countering depreciation of software assets. This was the foundation of Managing Software Debt: Building for Inevitable Change.

When we describe software only as a liability then we also provide reasoning to business that those who work on the software are part of a cost center. This is usually not the best way to operate any software development organization. The focus should be on optimizing the value that existing and newly created software assets support. Rather than only a liability, we can think of assets as what they are defined to be:

Assets are equal to “equity” plus “liabilities.”

The equity can still be there but if the liability is becoming so large that it outweighs the utility of the equity then we have a problem. We should counteract this decay in value by Managing Software Debt effectively as we produce new communications between the software and hardware assets we deploy.

Managing Software Debt in Practice Presentation

Today at the Scrum Gathering in Seattle, I held a session on “Managing Software Debt in Practice” where we got into:

The presentation had too much for the less than 90 minutes that we had for the session. I did not get into scaling Scrum team patterns and heuristics to manage software debt at scale and also less around testing than I’d hoped. Hopefully it was useful for the participants and they got at least one new idea leaving the session. It is difficult to take a 1-day workshop and create a less than 90 minute talk, as I learn again.

InformIT Interview with Chris Sterling

Matt Heusser, on behalf of InformIT, conducted an interview with me regarding the book Managing Software Debt: Building for Inevitable Change. We discuss what software debt is, some ways that it can be managed, maintaining a single list of work, how software debt is measured, and we even got into training and our product AgileEVM.com. You can read the interview at http://www.informit.com/articles/article.aspx?p=1684785.

Our Book is Available: Managing Software Debt – Building for Inevitable Change


I am quite happy about the book that took much of my time over the past couple years has finally come out. Thank you Addison-Wesley for asking me to write a book. Also, I want to thank Jim Highsmith and Alistair Cockburn for accepting the book into their Agile Software Development Series. Finally, I have to thank all of those that have guided, influenced, and supported me over my career and life, with special thanks to my wife and kids who put up with me during the book’s development. My family is truly amazing and I am very lucky to have them!

Movements of a Hypnotic Nature at AgilePalooza in Redmond, WA

Discussed the basis for high performing teams and the principles of Scrum as a framework , empirical process control mechanism, and the values. We then ran “Movements of a Hynpotic Nature” that helps us understand emergent design, iterations (if you only touch it once, you are not iterating), teamwork and operating from high-level requirements. This is based on cross-functional, self-organizing teams and overlapping development phases that are the roots of Scrum