Category Archives: Push-Button Release

Video of QASIG Presentation Focused on Managing Quality Debt

I had a great time speaking at the QASIG group meeting last night and met some great folks along with reconnecting with others I haven’t seen in some time. Here is the video of the presentation.


Video streaming by Ustream

And the slides

The Stability Index: Focusing on Release Stabilization

While recently working with Juni Mukherjee on a team that is focused on finding ways to extending and increasing the value of a large legacy platform she brought up what I thought was a brilliant idea. We had been working on creating metrics that have tension with each other to drive continuous integration effectiveness from the component level up to the deployed system and continue to generate value to users. Juni’s research and the team discussions on the topic went through multiple scenarios with different metrics to better understand how they balance and/or mislead. As we all know any metric can be gamed. And not only that, but a metric is not always valuable in a context even though it is extremely valuable in another. During the conversation Juni blurted out that what she was looking to come out with was a “Stability Index”. This brilliant phrase along with the outcomes of our team discussions lead me to think that this is a valuable way to look at quality measurements alongside other release constraints to support delivery of continuous value.

This article is a first attempt at putting The Stability Index down on paper as it is already in use, to some degree, in Juni’s organization. In the past, organizations and teams I have worked with have come up with similar approaches that allow us to balance effective quality signals. This should lead to early detection of what are usually weak signals of quality issues that many times are found too late or whose symptoms are seen away from the signals origin.

Goals of The Stability Index

Stability Index is a function of signals that are observed in pre-production and production environments. The purpose of calculating a Stability Index is two-fold:

  • The Stability Index is an indicator of how an organization is progressing towards its business goals. For example, if Stability Index goes up, Cycle Times reduce, Release Stabilization Period goes down and Customer Retention improves.
  • The Stability Index also reveals any correlation of pre-production signals to signals observed in production. For example, when Code Coverage goes up, the number of defects found in production goes down.

If proper balance is given to these pre-production and in production signals than this should lead to a stable application platform that continuous to deliver value at a sustainable rate as new business needs arise. Of course, this does not mean that business needs will be continuous or steady so there is still a potential to impact The Stability Index but by keeping balance, a team or teams should be able to manage fluctuations in business needs more effectively.

Ultimately, creating an implementation of The Stability Index is deciding on metrics that produce effective pre-production and in production early warning signals. The following sections will go into detail about the metrics which were are initially used in this implementation of The Stability Index.

Pre-Production Signals

Pre-production signals are focused towards technical craftsmanship of engineering teams.

Percentage of Broken Builds

This metric is used to objectively measure behavior patterns of engineers who may not be using effective gating criteria before checking in code. Build breakages could be while building source code, compiling, unit testing, distributing artifacts, deploying images or further downstream while testing for functional correctness, integration issues and performance gaps. Irrespective of where the breakage is, code will not be able to flow to production through a fully automated pipeline unless engineers test code changes in their local sandbox environments before checking into the common repository.

 Code Duplication

This metric is an early indicator of software debt that indicates that software programs have significant sequences of source code that repeat and this calls for refactoring. This has a high business impact in terms of maintainability.

Cyclomatic Complexity

This is a software metric that indicates the conditional complexity of a program (function, method, class etc.) by measuring the number of linearly independent paths that can be executed. This is also an early indicator of software debt and is very expensive to the company when it comes to getting new employees up to speed.

Code Coverage

Code Coverage can be either Line/Statement/Decision/Branch/Condition coverage and is a measure of how effective test suites are in certifying software programs. Code Coverage also indicates that 100% Line or Statement Coverage may give a false sense of security since all lines may have been covered by tests although important decisions, branches and conditions may not have been tested.

The caveat here is that a high code coverage percentage does not guarantee bug-free applications since the primary objective of tests should be to meet customer requirements and all customer use cases may not be exercised by tests even when Code Coverage is at 100%.

Test Cycle Time

Test Cycle Time is a measure of the time it takes to execute the entire test suite before either reporting bugs that would cause another iteration or software can be certified for production. For the same number of tests, the Cycle Time could be high unless tests are designed to be independent of each other and are launched in parallel. As an aside, for tests to be launched in parallel, the test environments can often become a bottleneck in terms of providing the required capacity and reliability. Virtual environments may not offer a guaranteed share of the CPU whereas shared clusters in distributed environments may queue jobs and hence make the execution time long and unpredictable.

Production Signals

Production signals are focused towards the customer.

Customer Delight / Satisfaction

It’s all about the customer. Software is released to meet the customer’s needs and to leave the customer delighted and craving for more. Although “Delight” can be subjective at times, surveys are an effective way to measure satisfaction. An example of customer dissatisfaction could be that albeit the software behaves correctly, it takes more number of clicks to perform the same activity than it used to take in the previous version. Defects reported by the customer are a good measure of this metric.

Defect Containment

This is an important trend to watch out for since customers can be inconvenienced if their support tickets are queued up. Moreover, defects reported in production that translate into code and configuration errors should be fixed by the engineering team within acceptable SLAs depending on the severity of the issues. Being able to iterate fast is one of the key factors for customer retention.


Systems could go down due to hardware incidents like router malfunction and disk crashes or due to software inadequacies like fault tolerance not being built in. Either way, downtimes cause revenue losses and are a critical contributor towards Stability Index.

Relationship of Signals to The Stability Index

Each of the above trends has a bearing on the Stability Index. Some trends are directly proportional to the Stability Index (), while the trends of others are inversely proportional (). For example, when Code Coverage goes up, it has a positive impact on Stability Index. On the contrary, when Code Duplication goes up, Stability Index goes down.

The relationships of all the metrics with Stability Index are illustrated below.

Pre-production Metric Relationship to Stability Index
Percentage of Broken Builds  
Code Duplication  
Cyclomatic Complexity  
Code Coverage  
Test Cycle Time  
Production Metric Relationship to Stability Index
Customer Delight / Satisfaction  
Defect Containment  

Recognizing Software Debt Talk at Beyond Agile Meeting

A couple days ago I spoke at the Beyond Agile group meeting on the topic of “Recognizing Software Debt”. Early in the presentation we ran an exercise to get a feel for the effects of software debt that was original created by my friend, Masa Maeda. Here is a link to the exercise:

The exercise went great even though I was using the audience as guinea pigs to run the exercise for my first time. Below is the slide deck that I used as a backdrop for the presentation.

SeaSPIN Talk: Dollars and Dates Are Killing Agile

Last night I spoke at SeaSPIN on the topic “Dollars and Dates Are Killing Agile”. The focus of this talk is how we can speak more like business with the benefit of building organizations that are more supportive of adaptive planning, continuous improvement and team empowerment. If we do not speak the language of the business and continue to create friction without enabling strategic planning needs that businesses have then we are bound to see Agile methods have a short lifespan in most organizations. Below is the topic description and slides:

Agile teams speak in points and iterations, but project and business managers think in terms of dates and dollars.  This conceptual and language barrier makes strategic business planning, funding, and project status reporting a significant challenge for Agile teams.  Because of these barriers, many successful Agile/Scrum initiatives are discontinued or never expanded.

Delving into Technical Debt – Cutter Article

The following is an except from the article authored by Israel Gat and myself named “Delving into Technical Debt”:

Many of the findings and the recommendations we make in Cutter technical debt engagements are broadly applicable in concept, if not in detail. There is commonality in the nature of the hot spots we typically find, the mal-practices we identify as the root causes and the ways we go about reducing the “heat.” Granted, your technical debt reduction strategy might dictate investing in automated unit testing prior to reducing complexity, while your competitor might be able to address complexity without additional investment in unit testing. However, the considerations you and your competitor will go through in devising your technical debt reduction strategies are fairly similar.

It is this similarity that we try to capture in this Executive Update. Some of specifics we recount here might not be applicable to your environment. However, we trust the overall characterization we provide will give you, your colleagues and your superiors a fairly good “3D” picture of how the technical debt initiative will look like in the context of your own business imperatives and predicaments.

As a Senior Cutter Consultant, it has been a pleasure working with Cutter to release this Executive Update from which the excerpt cited above is taken. For a free donwnload, click here and use the promotion code DELVING. Let us know what you think about the article in the comments section of this post.

Puget Sound PMI Talk: Integrating Quality into Project Portfolio Management

Last week I did a talk in Lynnwood, WA with the Puget Sound PMI chapter on “Integrating Quality into Project Portfolio Management”. I feel that the slides were the best yet in providing specific indicators and understanding around the troubles with scaling Agile methods and in strategic decision-making at scale. Although stage, phase, or “gated” approaches don’t provide a better answer than Agile methods, they provide an illusion of knowledge and control that Agile methods do not initially bring to the table. This talk goes into patterns for scaling Agile across an organization, the issues inherent in scaling, and ways to identify, specifically from a quality trend perspective, if there is troublesome waters ahead on specific project endeavors. The focus is on those quality indicators being found amongst all the noise in scaled projects to figure out if the project should be re-committed to, transformed, or killed (reference from Johanna Rothman in her book “Manage Your Project Portfolio”) so that value can be optimized.

So, without further ado, here are the slides…

Interview with Spot On Business: Agility for the Long Haul

On November 1st at 2:00pm ET (11:00am PT) for about 45- to 60-minutes, Gil Broza will have me on as an interview guest on his series “Spot On Business” talking about “Agility for the Long Haul”. You may sign up at to be part of the conversation. Here is a brief description of the interview topic:

If you’ve already started using Agile methods, surely you’ve seen just how much change and adaptation are involved — already at the single team level!
If you’ve already applied Agile methods across a program, or several unrelated teams, you’d have noticed that their performance is constrained by downstream and upstream functions. Your teams are likely saying, “We need marketing and sales and the business units and operations to also act like us in order to really make an impact.”
And if you’ve been using Agile for more than a couple of years, is performance still great, or is it eroding? Are the Agile mechanisms slowly giving way to “business as usual”?
Please come and join us for the interview.

Treating Software as an Asset

William Caputo wrote a passionate blog entry on why “Software is not an asset” here:

Although I entirely agree with ideas discussed about refactoring and removal of code, I do not think that the blog entry substantiates his claim that software is not an asset. An asset is:

Anything tangible or intangible that is capable of being owned or controlled to produce value and that is held to have positive economic value is considered an asset

Much of the software out there has a positive economic value either in productivity, revenue generation, cost reduction, or other types of economic value. I see removal of code and refactoring code into a more desirable design in order to support changing business needs as essential to countering depreciation of software assets. This was the foundation of Managing Software Debt: Building for Inevitable Change.

When we describe software only as a liability then we also provide reasoning to business that those who work on the software are part of a cost center. This is usually not the best way to operate any software development organization. The focus should be on optimizing the value that existing and newly created software assets support. Rather than only a liability, we can think of assets as what they are defined to be:

Assets are equal to “equity” plus “liabilities.”

The equity can still be there but if the liability is becoming so large that it outweighs the utility of the equity then we have a problem. We should counteract this decay in value by Managing Software Debt effectively as we produce new communications between the software and hardware assets we deploy.

Towards a Push-Button Release

This presentation is being delivered as a 45-minute lecture and discussion for a company-wide tech talk today. It contains 2 case studies that revolve around moving to a push-button release that reduced the whole product company’s release cycles from 6 months to every week and the effects of a “No Defect” policy on a team’s productivity.