Submitted by Jamal Moustafaev on Tue, 11/17/2015 - 11:36

Why Your IT Project May Be Riskier Than You Think

I have recently stumbled across a very interesting article titled "Why Your IT Project May Be Riskier Than You Think" published by Bent Flyvbjerg and Alexander Budzier.

I am a big fan of professor Flyvbjerg and his work and as a result needless to say I read this article with great interest. The article describes the phenomenon called the "black swan projects" or as he puts it "high-impact events that are rare and unpredictable but in retrospect seem not so improbable". In other words, these are the endeavours that are vastly over budget and sometimes so bad, that they kill the companies that conceived them. Some of the examples are:

Levi Strauss' migration to the SAP system (original budget - $5 million, actual cost - $192 million)
Hong Kong’s airport's IT system upgrade (losses of $600 million)
Hershey’s a new order-taking and fulfillment system (losses of $100 million causing an 18.6% drop in quarterly earnings)
Kmart's IT modernization project (original budget - $1.4 billion, actual cost - $2 billion; led to company's bankruptcy)

Professor Flyvbjerg further states:

"The average overrun was 27%—but that figure masks a far more alarming one. Graphing the projects’ budget overruns reveals a “fat tail”—a large number of gigantic overages. Fully one in six of the projects we studied was a black swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. This highlights the true pitfall of IT change initiatives: It’s not that they’re particularly prone to high cost overruns on average, as management consultants and academic studies have previously suggested. It’s that an unusually large proportion of them incur massive overages - that is, there are a disproportionate number of black swans."

The article proposes the following steps to be taken in order to address this challenge:

The company should be prepared to deal with the situation where its biggest technology project goes over budget by more than 400% and only 25 to 50% of the projected benefits are realized
The company should also get ready to the situation where 15% of its medium-sized projects go over budget by 200%
Break the behemoth projects into smaller more manageable ventures
Use reference class forecasting (in a nutshell "what was the mean cost and more importantly standard deviation on similar projects undertaken by other companies") as much as possible.

My Contribution To The List

As a project manager with approximately 20 years of experience in IT and software development industry I must say that I wholeheartedly agree with these conclusions, but my experience tells me something else, something very important, is missing from the list above. My fifth ingredient to the mix would be:

Invest more time in requirements elicitation, analysis and documentation even before the project has started.

In other words, do some planning before the project initiation. Let me explain my point of view. Several years ago I had a chance to conduct a spontaneous experiment in Dubai. I was teaching my Project and Portfolio Management Masterclass . The room was full of professionals predominantly from the construction, engineering and telecom industries and at some point of time we got talking about estimation on IT projects. I proposed to ask them two questions just to demonstrate how different human mind works on perceiving the sizes of IT and non-IT endeavours.

Example #1: Burj Khalifa vs. Amazon - The Iceberg Phenomenon

Q1: Look out of the window. Do you see that large building outside? How much do you think it cost to build Burj Khalifa (currently the tallest building in the world)?

A1: Oh! At least several billion dollars! (Actual cost $1.5 billion)

Q2: Have you ever used Amazon or eBay? What is in your opinion the total cost of development of their platforms?

A2: Couldn't be more than U$100,000 or 200,000 (actual cost are in the neighborhood of tens of billions of dollars)

Q3: (after disclosing the real costs of the projects) Why did you successfully estimated the cost of Burj Khalifa and totally missed with your estimates for the IT projects?

A3: Well, man, look at the size of this monstrosity! It had to be somewhere around a couple of billion ... As to the websites ... You logon, browse the catalog, add something to the basket and pay for it. How complicated can that be?!

This conversation was provided in order to illustrate why I very frequently refer to the chronic underestimation of the scope of the IT projects as "iceberg phenomenon". The problem is that an average person fails to see the complexity of any software development initiative because he/she only sees the very top layer - the interface of the product, while all the infinite lines of code required to make sure the interface works properly remain hidden to them.

Example #2: Airline Kiosk Requirements - The Crack Spread Phenomenon

By the same token, I sometimes refer to the requirements elicitation process as a "crack spread phenomenon". Let me explain what I mean by that.

I frequently like to play the following trick on my course audiences. I ask them whether they ever travelled on an airplane. Obviously almost all of them say "yes" and the following dialogue takes place:

Q1: Have you ever used the airline kiosks that dispense the boarding passes?

A1: Yes, of course!

Q2: Do you think this was an easy or a difficult task to create these kiosks? Note, that I am not referring to the hardware, but to the logic that went into the step-by-step system.

A2: We think it was easy. After all you just swipe your passport and the system prints your boarding pass. How difficult could this be?

In reality here is a list of questions a professional requirements analyst would ask just regarding the passport identification procedure:

How many types of passports are out there? (number of countries in the world multiplied by the types of passports per country)
Do all passports in the world follow the same encoding standard?
- If not, then how many standards exist?
What happens if the machine is unable to read the passport?
If the machine is able to read the passport what happens if:
- Passport is real and not expired?
- Passport is real but expired?
  - Will the user be issued a boarding pass if this is an international flight?
  - Will the user be issued a boarding pass if this is an internal flight?
- Passport is fake?
  - Should the kiosk notify the police?
    - Does this mean that every device needs an interface with an airport police department?
    - Does this imply that we need to deploy some kind of software in the airport police headquarters?
  - What should the kiosk “do” while the police if being notified?

Note that I haven't even gone into the discussion of other types of identification (credit cards, frequent flyer cards, etc.), finding the reservation, different options available for frequent vs. one-time flyers, meal requirements (Kosher, Halal, Vegan, etc.), baggage payments to name just a few.

Did you notice how the scope of a seemingly simple project spread like the cracks on the windshield? One question leads to a new problem, which in turn leads to several more questions requiring unequivocal answers. All one needs to do right now is to imagine how many similar "cracks" will appear when deploying an ERP system at a global manufacturer, or a payments system at an international bank, or core billing system at a large wireless provider.

Conclusion

In order to prevent your next large project from becoming the proverbial "black swan", in addition to the steps outlined by professor Flyvbjerg we need to conduct a preliminary requirements elicitation exercise before the project initiation in order to correctly assess the project scope and potential risks. One does not, in my opinion, go very deep in this exercise; my guesstimate would be to go at "Feature plus high-level requirements or questions" just like was illustrated in the "Airline Kiosk" example.

This will serve two purposes:

Establish at a much higher level of detail project scope and risks (and the resulting time and resource estimates) and
Prevent the biased executives from using the Machiavelli Factor allowing them to artificially inflate the benefits of the project while simultaneously deflating the project size and complexity.

About the Author

Jamal Moustafaev, MBA, PMP – president and founder of Thinktank Consulting is an internationally acclaimed expert and speaker in the areas of project/portfolio management, scope definition, process improvement and corporate training. Jamal Moustafaev has done work for private-sector companies and government organizations in Canada, US, Asia, Europe and Middle East. Read Jamal’s Blog @ www.thinktankconsulting.ca