November 7, 2013

The Autopsy

Disaster rarely occurs without lots of inside people knowing it is coming. The faulty bolts on the San Francisco Bay Bridge, the California Unemployment Insurance payment debacle, and web site disaster of the new Affordable Care Act known as Obamacare are a few glaring examples of huge system failures currently in the news.

Officials from the Employment Development Department (EDD) are currently being reamed by California lawmakers for a glitch on a new computer system that has disrupted the delivery of benefit checks for almost 150,000 unemployed Californians. Deloitte Consulting is taking a huge hit on the bad press generated by the debacle, as are the EDD executives.

Barack Obama's approval rating has dropped to the lowest level, with the centerpiece of his presidency, Obamacare, now a joke on late night television... "Obama said they've had some glitches with the Affordable Care website. I'll tell you something. If you order a pair of pants online and they send you the wrong color, that's a glitch. This is like a Carnival cruise, for God's sake!" (David Letterman)

32 large bolts, 9 to 17 feet in length, snapped on the new east span of the Bay Bridge. Many more failed to tighten properly. The failure is due to excessive hydrogen in the steel. This engineering failure is huge! The new span cost $6.41 billion. More than a quarter of a million vehicles cross the Bay Bridge everyday! Early estimates are that it will cost more than $5 million to fix the bolt problem.

It is hard to believe that no one knew about these problems long before the press arrived, before it caused major pain to the public and heart burn for top administrators and executives. Where was the oversight on the healthcare site? Where was the quality testing on the EDD computer system? Where was the quality control on the bolt manufacturing? The public is left to wonder – who is looking out after the public interest, and how safe will the bridge be in a high magnitude earthquake, the kind of disaster that the new span was designed to handle.

It is hard to fix a problem when the failure becomes a political football being kicked all over the field. No one wants to own the problem. Heads roll and careers are ruined.

Payson Hall, is a Project Management Consultant with Catalysis Group, a Sacramento firm that specializes in software and systems project reviews and oversight - autopsying major software project disasters. According to Hall, "It is rarely, if ever, the case that the troops in the trench don't know there is a problem." But it is risky at best to share the bad news. "Sometimes management is hiding the ball but sometimes they really don't know there is a problem." The IT staff usually knows when a system is going to fail – the programmers, the software engineers - the techies working on the project. "It isn't going to deliver on its promises and the team knows it."

By the time the undertaker arrives the disaster is in the public eye and everyone in a position of power is stuttering – unable to explain the mess. Many who actually do the work in a large institution are far removed from the management personnel at the top who are in a position to stop or delay the project until a solution to the problem can be found.

The whistle blower dilemma is who is going to tell the emperor he has no clothes on? No one stands up to speak truth to power. If you do, common wisdom is that you will be destroyed in an instant. Countless whistle blowers have faced retaliation for reporting on government waste, fraud, and abuse. It is not supposed to happen but it does.

Hall believes that no one told the president there was a problem on the healthcare web site. "Of course Barack Obama had no idea the Affordable Career Act web site was going to fail so spectacularly. He would never have gone on national television touting the web site. If he had known, he would have surely delayed to roll out" says Hall.

In his article Escaping the Clutches of Zombie Projects, in STQE, the software testing and quality engineering magazine Hall says, "When you believe that there are negative consequences for delivering bad news, it takes a lot of courage to speak up when something's wrong. Telling people "Don't be afraid" is hollow advice in those organizations that make a habit of shooting the messenger." "Trouble arises when organizations reject any discussion of actual or potential project problems as "negative thinking." The shift can be gradual and subtle, but the consequences are devastating."

The day of reckoning on a problem eventually arrives. In the case of huge public works disasters, heads roll and careers end. No one wants to be the bearer of bad news- no one wants to carry the water.

Until the gods are willing to listen and there really is protection for telling truth to power, we will autopsy the dead project and we will continue to have disasters that cost dearly and could be avoided.