Will Our Software Bankrupt Us?
Recorded January 29, 2019
In this interview, Jeff and I talk to Herb Krasner about his recent study on The Cost of Poor Quality Software in the US. We ask him about what lead him to this research, he walks us through some the of key insights and we discuss if the notion that we may not be able to afford the software that runs our business.
Listen to the podcast on Spotify
Read the transcript:
Pete Pizzutillo: Herb thanks for joining us today. But before we get into our discussion on the Cost of Poor Quality Software, you have a rich background in software engineering and research, can you walk us back as to how you ended up here?
Herb Krasner: I started my professional career as a professor of computer science at Clemson University. I left academia and got involved in fixing troubled software projects. That led me to join the microelectronics and computer technology consortium where we started a program on empirical studies of software engineering.
I consulted and worked in many large companies trying to help them improve the way that they develop and deliver software. During that time, I was introduced to the capability maturity model and helped companies improve with the CMM approach. Though while doing so, I also started to collect data on the cost versus the benefits of good quality.
In the ’90s I published a new version of the cost of software quality model, and it was that work that specifically led me to this new report. I became a professor of software engineering at the University of Texas, and I did some research on the cost of software quality. But my experience has been that going into companies and helping them improve the way that they develop and deliver software; I always encounter the question especially from the execs, what is this going to cost us and how’s it going benefit us?
About the Cost of Poor Quality Software Report
Throughout my career, I collected data on precisely that, so I had cost-benefit data, for example for improving using the CMM model from level one to level two to level three and so forth.
About a year and a half ago, I was asked by a client if I would collect data and report on what is the magnitude of the cost of quality in today’s world. I immediately thought of CISQ, and if I do it, I should do it under the CISQ banner because it naturally fits with the mission of CISQ looking at IT software quality.
Pete Pizzutillo: There are a lot of industry reports published on the cost of security breaches as security has clawed its way to the top of the quality conversation over the last fifteen years or so. When there’s a breach, we can quantify those costs, but there are very few people, you might be the only person I know of, that’s looking at software quality in a broader definition and trying to pull it apart to say, here’s the epidemic. It is much more significant than just security. And that’s why I’m excited about our conversation today.
Herb Krasner: Well security breaches are the hot thing, and they get a lot of attention, which is good, I think if we direct that attention into improving quality in general, I think that would be something good.
Jeff Fraleigh: So, the market is looking for information around software quality which is refreshing, because, if you look at DevOps and Agile, we’re very eager to identify that we need to do things faster, better, and cheaper. We change the process and go through re-education and re-tooling without really understanding the current state, quantifying the current state to then determine if we got better. I don’t know if you’re getting into those questions.
Herb Krasner: I’ve been involved in those questions ever since I started consulting with a variety of companies.
Pete Pizzutillo: But before we get to that, do you want to give us a little background on the report?
Herb Krasner: What I tried to do, was to see what existing sources of information were out there that I could draw from because I didn’t have the resources to do an entirely new in-depth study where I sampled lots and lots of organizations and so forth. I started searching for things, and I found little pieces here and there. For example, one of the interesting reports that have been done is this thing that Tricentis does on software failures, so you know they’ve been highlighting the software fail report which is just those things that show up in the news that get quantified. That’s a piece of it, but that’s the tip of the iceberg. I wanted to get well below that and look at all those projects and software that is being abused and misused in organizations that never get reported, so I started to look at it differently.
And that led me to looking at the world from what’s happened in the past, where we are now with the current crisis, and where we might be in the future. Taking that orientation, I was able to tease out things like, looking at legacy systems as a category and that looking to see what information was out there about the problems in legacy programs, and how much is that costing us.
That’s all stuff which is unseen from the public because it’s all happening inside of IT organizations inside of companies. And yet that’s where all the dollars are spent. When you ask the question, from the overall perspective of the total number of IT dollars that are spent, you know, in the world and the USA today, 75 to 77% of that is being spent on legacy system care and feeding, so why not look at that part of the problem. That’s where all the money is. So that was one of the reasons for carving out that area and digging around and seeing what I could find about that, and it turned out to be a significant piece of the total puzzle.
Breaking Down $2.84 Trillion
Jeff Fraleigh: That number jumps out. I know you and I are going be talking soon about just the impact of the application modernization, but this goes directly to that, right. I think people are modernizing for things like speed and agility, but apparently there is such a financial ring around the neck of the technology world. There are just so many legacy applications out there that are costing people money, and these numbers are mind-boggling, it was the first thing that caught my eye your 2.84 trillion-dollar number in terms of what is the cost of software quality. How do you even put that into context? What did you think about that when that came up when you saw that?
Herb Krasner: When I arrived at that number, it looked enormous to me. Though my intent was not for it to be an exact number but to be a first-order approximation that we could then start from and work our way to something more detailed or exact. So, I don’t consider that to be a number that’s worth citing as fact. It’s an approximation.
But when you compare that number to, US GDP, it’s a considerable portion, if that’s anywhere close to being true, it says that we’re spending 10% of our GDP on the cost of poor quality software. And that’s amazing.
Jeff Fraleigh: I think whether it’s $2.84 or $2.54 or anything in that realm it’s still gigantic, and I did that same translation around what does that mean against our GDP. I know that you broke it down into sections, but where do you think people are the most blind when it comes to impact, is it strictly in legacy as you mentioned? Is there anywhere else where you feel like this adverse effect is hidden?
Herb Krasner: Well, it’s apparently in legacy. It’s also in all the effort spent finding and fixing defects in software that is not fielded yet. And that is hidden inside development organizations, and there’s a lot of money spent on that, at least according to experts like Capers Jones who has looked at that in some detail.
It’s also the case that most of the projects that are in trouble because they’re over budget and behind schedule and delivering poor stuff, most of those projects are not seen outside of the organizations or outside of the companies in which they’re happening. So that’s another area where a lot is happening, and the Standish Group has tried to highlight that area, where troubled and canceled projects are going on inside of organizations that don’t reach the news. And therefore, people don’t see or know about. So, between the problem of finding and fixing defects inside of the development organizations before things are delivered and troubled and canceled projects, those things start to be very large numbers.
Then if you throw in the idea of technical debt, that we’re mortgaging our future by creating poor quality software that can’t be evolved and modified, that is well over a third of the number. So all those areas are pretty much unseen.
The only thing that is seen is what hits the news, and what hits the customer base. When the customer tries to use an app on their phone, and it has a problem, it blows up or something. Those are all things that are seen by the general public, but most of the stuff that we’re talking about here is entirely unseen.
Software’s Impact on Projects and Project Teams
Pete Pizzutillo: So, you are breaking down that number into a couple of categories which I feel are easily connected to poor quality software. Legacy systems, for one, failures, security defects. Technical debt is a description of how much poor quality is in there, but troubled and canceled projects, can you help us think through how you see poor software affecting the delivery of projects? Is it just technical debt, is it something else?
Herb Krasner: It’s not technical debt, it’s a complete waste of resources. You remember when we talked about this idea that if 75% of all the total dollars were being spent on legacy applications that leaves 25% for development. Ifa good portion of those development projects are in trouble, then that’s a huge waste of scarce resources that are needed to develop new things and make new things that are of high quality. Even though it’s hidden, it’s costing somebody.
And the report points to a couple of projects that I was aware of, and that I had been involved in, that are hugely costly. For example, as a taxpayer of the state of Texas, I’ve got a project out here that I was consulting with that’s way behind schedule and over budget and not delivering. That subtly impacts me but impacts the agency that’s trying to provide the services that that software system is based on because it’s inefficient. Using 25-year-old technology, they’re not going to be able to accomplish their mission effectively. There are subtle things that are impacting us in one way or the other, and we’re all paying for it.
Pete Pizzutillo: Jeff and I have spent some time talking about Agile and DevOps. There are some shining stars out there in terms of the cloud-native companies that are delivering software at mind-boggling speed. So, given all these process improvements, all this innovation in the technology and the ability to automate the development and delivery chain. Aren’t we getting better? Is there any good news along the delivery cycle that you’ve seen along your travels?
Herb Krasner: You cited a couple of things, which are techniques, technologies, process improvements, practices. But if you look at Agile. Agile has helped deliver features faster, but has it helped deliver features that are necessarily better? Sometimes yes, sometimes no. I’m aware of a lot of organizations that are misusing Agile and delivering crap because they can do it faster and cheaper.
There’s a dark side to each of those things that you mentioned. As far as DevOps is concerned, and when you pair it with things like continuous and early testing, yeah. We’ve seen some improvements there.
I think it’s like anything else, which is a process improvement or a new practice. There’s a bright side and a dark side. These things need to be done correctly, and with discipline. It boils down to good ole software engineering discipline, where quality is, in fact, a goal rather than just delivering something faster and cheaper. My view of the world says, there’s a triangle.There’s cost, there’s schedule, and there’s quality. And if you don’t pay attention to all three of those things, using whatever processes you’re using, then you’re not going to get quality. I still think it’s the case that there are not enough people, especially not enough executives that are paying attention to quality.
And that’s one of the reasons why this report is important to me, because I if I could take this report and I could take it to every CEO in every large company and beat them over the head with it, I would do that so that he would start asking questions.
What Should Organizations Do Today to Address Poor Quality Software?
Jeff Fraleigh: So, Herb, what do you hope the outcome is? Let’s say you get to the C-level, and they’re able to read this, what steps should they be taking to help their organizations with this problem?
Herb Krasner: I recommended some simple questions that they should start asking, and that’s hopefully what this report has communicated. That asking the question “what is the cost of poor quality software in our organization today?” Just asking that question, is revolutionary, how do our investments in quality affect our overall cost of quality and the cost of ownership of our software intensive assets? Those simple questions will change the entire culture of an organization. I’ve seen it because I’ve worked with organizations who I have convinced to start asking that question. What it drives is then the organization realizes that software quality is essential. And that they need to start measuring software so that they can talk about software quality in a meaningful way. And that there are costs, and benefit implications associated with it. So, if I could influence behavior, it would be merely to get CEOs to ask that question.
Pete Pizzutillo: And I think you present something in that statement. Quality is a loaded term, and Good is even more loaded. I believe there is some help to say, what do we mean by good software? Can you talk a little bit about that?
Herb Krasner: Sure. There’s a general meaning and a more specific meaning. Software quality is one of those things that you might say, it’s in the eye of the beholder, like beauty. It’s something that’s not very well defined. But when you get down to it, some standards organizations have more precisely defined software quality. So even though in general it might mean different things to different people. In organizations that I work with, I encourage them to define it accurately.
Generally, the idea is that a piece of software satisfies its users and customers and doesn’t cause any harm and something general like that. If you look back at Crosby and others who’ve been quality experts in the past, they talk about things like conformance to requirements, fitness for use, meeting company standards, or external standards in some way. Structural quality, as Bill Curtis will talk about in more detail. And things like aesthetics quality. Those are all more specific parts of what we mean by quality.
But even further than that, we have organizations like ISO, who have gone to the trouble of starting an industry-based standards project to try and define very precisely what we mean by software quality, so. ISO 25000 is a standard that defines what exactly we mean by software quality and beyond that, how do we determine the individual characteristics and how do we measure them.
CISQ has tackled several of those more specific aspects of what indeed we mean by quality. So, there’s the general definition and the organizational-specific definition. There should be a project-specific definition for each project so we can determine priorities amongst the different aspects of quality and then there’s the industry standard, like ISO 25000. We have definitions of what this concept of software quality should be, and we need to pay attention to those.
Pete Pizzutillo: We started the conversation asking the question ‘Will our software bankrupt us?” and through your journey and research it’s apparent that it is bankrupting us. And your counsel to business owners is to start asking the questions because each company’s different, so ask is our software bankrupting our company? It’s a significant first step, and I think you’re laying out a path, stop arguing over the definition of good and safe and sound, adopt a standard, pick it. There are a lot of smart guys working on that same problem. Start measuring software and software quality to understand where you are and then get better. These are good recommendations for business leaders that you laid out there.
Herb Krasner: Absolutely and there are organizations again, like CISQ, who are developing standard measures and tools to do those measurements, so we don’t have to burden the organization with a lot of data collection if we don’t need to, if there are specific metrics and tools that are going to be available.
Pete Pizzutillo: So, what’s the next step for your journey? I mean, you’ve been going on for a while, different things. Is there a follow-up or a deeper dive, do you see any aspect of this research coming up?
Herb Krasner: Well I would love for there to be a follow on. I haven’t talked to anybody about that, but I could foresee doing another version of this report, maybe yearly or a bi-yearly basis. I think it … I guess what I’m waiting for, is to see what kind of reaction that I’m going get from this report.
It’s published on the CISQ webpage, and there were a lot of downloads, a lot of people that have read it, but I haven’t seen much in the way of feedback, to be honest with you. I welcome input from your audience to give me some idea of whether it’s worth pursuing this regularly. If it’s worth creating more in-depth studies to tease these out and try to come up with not only some idea of exactly what the amount is, but is it growing or shrinking, what’s the trajectory for the future? Again, back to your question of is this software going to bankrupt us eventually? Well, it’s certainly making us vulnerable, that I can tell you.
Herb Krasner: I’d love somebody to, if they tell me I’m crazy, to give me something else to look at, that will refute these huge numbers that I’m coming up with.
Jeff Fraleigh: Thank you Herb. Everything that you’ve done is terrific, the numbers are mind-boggling and frightening and scary. But people and companies need to know, and we’re happy to be a small part of getting this information out to more people, and I do encourage and hope that some people give you feedback. But this has been great, thank you so much for your time today. This has been great.
Herb Krasner: It’s been my pleasure, and hopefully if I pursue this more deeply and we go into the broader subject of legacy, you’ll bring me back for another session.