Tracie Berardi, CISQ Program Manager

Software risk has historically been overlooked as a security concern by business leaders, and companies have paid a high price as a result. Remember the debacle earlier this year when HSBC services went down, leaving customers unable to access their online banking? That was during the peak of tax season, causing a flurry on social media and deeply damaging the company’s reputation with customers.

Companies have also had to dish out high sums to compensate their customers. RBS paid £231 million for their IT failures a few years ago, and the Target breach cost the retailer $152 million in addition to chief executive turnover. Most recently, Jeep controls have been taken over by hackers, and a similar incident with Toyota-Lexus leaves the manufacturer fixing a software bug that disabled cars’ GPS and climate control systems.

Poor structural quality of IT systems and software risk are not just IT issues. They are big problems that can lead to lost revenue and a decline in consumer confidence. So I was thrilled to know that the topic for the annual Software Risk Summit in New York was indeed just that, software risk.

Panel guests from BNY Mellon, the Software Engineering Institute at Carnegie Mellon, the Boston Consulting Group and CAST shared interesting “real world” insights. But beforehand, I was able to sit-in on the keynote by Rana Foroohar, who is a regular commentator on CNN news and a global economics analyst for TIME Magazine, among others.

Rana made a very important connection between America’s post-recession recovery and the role software risk will play in companies’ ability to create real, sustainable growth. According to Rana and her book Makers & Takers, we are entering a period of volatility with lower long-term growth, an unstable U.S. election cycle and a growing wealth divide. Because of this, the private sector is going to take on a bigger role in turning technology and infrastructure into tangible value that will carry the country through a period of “public sector slump.”

She shared an interesting statistic, noting that pre-2008, companies and consumers held the majority of the country’s debt. Now that paradigm has shifted, with consumers and corporations becoming more debt-averse, leaving the U.S. government to carry the vast majority of our debt burden. In this coming era of increased dependence on the private sector to create and sustain a thriving economy, it is more important than ever for business executives to take software risk seriously, take stock of their technology investments and prepare for future waves of innovation.

Following Rana’s inspiring keynote, the panel discussion dove-in head-first to the tactical application of software risk mitigation. Here is a brief summary of the interactive Q&A:

WHY IS SOFTWARE RISK A PROBLEM?

Benjamin Rehberg, Managing Director, BCG: The biggest responsibility lies at the CEO and board level. Many leaders may realize they’re becoming a technology company, but they’re not quite sure what to do about it. Most CEOs want to focus on boosting revenue, but they fail to recognize technology as a strategic enabler of the business.

Early technology was originally used to run internal systems, so the incentive for developers to write resilient code was very low. Only 20 years ago with initial exposure to the Internet did we start to see the need to worry about risk in systems that are directly end-customer facing. So there’s still a lot of digital risk buried in millions of lines of code.

However, with the increased publicity of big software glitches, there is more pressure to keep the business running and customers satisfied. For example, board members and CEOs are starting to think about what will happen to them if big security issues and breaches continue to plague their companies. Their company performance and jobs are at stake, so personal incentives are becoming more important and are starting to drive change.

Kevin Fedigan, Head of Asset Servicing and Broker Dealer Services, BNY Mellon: Leadership must take a progressive attitude toward risk and treat it as a core organizational value. For example, BNY measures three levels of risk: 1) general employees, 2) traditional compliance roles, 3) internal and external auditing. The financial services industry, in particular, has a reputation to uphold. We need to ensure customer trust in our systems.

Dr. Paul Nielsen, CEO of SEI: Some CEOs are uncomfortable with risk, so they delegate it to their CIO. But even then, they can’t rid themselves of the responsibility. This creates more of a stigma around risk and fosters an environment where it can grow and lead to bigger problems down the line. It’s interesting to see us all rushing to the Internet of Things, but most of the technology supporting this shift was designed with code written before the Internet. We clearly still have some catching up to do.

Vincent Delaroche, CEO of CAST: This may seem like a paradox because there is such high demand for security, but the root cause of many software catastrophes are actually resiliency and efficiency issues – not security flaws. Security gets the glitz and the glory, but the press sometimes misses the true root cause of many software issues, thereby misleading executives to search out security tools rather than solutions that help with resiliency and efficiency. I believe we are reaching a tipping point where there will be a spoke in demand by the Fortune 500 to assess their real risk exposure.

WHAT DOES CULTURE HAVE TO DO WITH SOFTWARE RISK? DO WE HAVE A COMMUNICATION ISSUE? WHAT IS IT NOT DOING TO GET THE BUSINESS AND THE BOARD’S ATTENTION?

BNY Mellon: We make the business own the risk, so risk is not removed from business outcomes. For example, with high priority items, that risk must be removed within a 30 day window. Our CEOs report to the Chief Risk Officer to ensure we aren’t putting risk and security to the wayside. We’re doing the best we can to remove those communication barriers and increase transparency between operations and the business.

SEI: There are too many risks to address them all, so you have to figure out what really matters. By setting benchmarks, it’s easier to measure your investment and prove ROI. There is a set of specific risk issues that have been identified as important by the Consortium for IT Software Quality (CISQ), an organization we’re involved in, along with a standard measurement framework.

BCG: Financial risk is a big topic for many of our clients. Financial institutions are constantly being hammered by regulators to comply, but they have such a broad range of technologies to manage. Because of this, we’re seeing that most outages are actually due to the “plumbing in between” various components of core systems and business processes. Very few technologists are actually looking at the transaction level between the technologies, and that’s a big place we see clients get messed up.

WHAT IS THE CORRELATION BETWEEN COSTS AND RISK?

SEI: Effective leaders will balance cost with productivity. It’s important to determine what is vitally important to your business and make sure those systems don’t degrade, but you must also prioritize where the investment goes. Many leaders still don’t understand where the risk comes from. The industry would benefit from a “genetic testing of software.”

BCG: The earlier you catch risks, the less it will cost to repair. What we’ve seen work well is creating a culture of incentives for high quality code. There’s a big push for IT organizations to become more agile and set up code peer reviews to help create more robust software.

CAST: Let’s say a development team for a large enterprise has about 1,000 engineers. Each year, an IT department this size will have to deal with about 20,000 software defects in production. When we look at these defects, we typically see that 90% cost very little, maybe a few hundred dollars. 9% of defects cost the business an average of $5,000, and only 1% of defects are severe enough to cost the business upwards of $50,000. So, individually, these errors are small. But when we look at them together, we see enterprise CIOs writing off upwards of $20 million per year and not thinking twice.

Conversely, if you look at the top 1% of the 1% of severe defects, this is where you see the massive breaches and glitches that sometimes end up in the press (like RBS, HSBC and others). These outages can cost companies an average of $600 thousand, according to the most recent KPMG Risk Radar, and very quickly catch the attention of senior leaders and CEOs.

At CAST, we help illuminate and prevent the 1% catastrophic risks and some of the “hidden” costs of the 99%, showing CIOs how to get more from their IT departments. Left unchecked, these common issues can consume more than 20% of ADM budget and keep developers from focusing on delivering new, innovative value to the business.

The good news is that we have concrete data points and studies that show the correlation between product defects and software flaws. There are currently about 60 critical flaws documented by the Consortium for IT Software Quality that need to be addressed, so this is manageable for CIOs and IT departments. And, the same set of flaws that reduce the risk of newsworthy incidents also lower the unseen cost of glitches.

As the Software Risk event would indicate, it’s clear that some companies are leading the way forward by integrating IT innovation with strategic business outcomes. But many are still stuck trying to justify IT expenditures that don’t necessarily correlate to growth. Organizations that link software risk performance with executive objectives will fare better than others.

If history tells us anything, it’s that there is only a matter of time before another cataclysmic glitch takes down core banks, exposes consumers to identity or credit card theft, and costs corporations millions of dollars. Establishing an effective software risk framework will pay off in dividends.