“Mutiny” On The High Seas
In 1707 a British fleet under the command of Sir Clowdisley Shovell was returning home after a long mission. The convoy encountered a heavy fog upon entering the English Channel. The admiral gathered his officers and navigators to get a definite fix on their location. After a short consultation the officers reported to Shovell that the fleet was safely off the coast of France. The admiral was about to issue an order to sail North to England when a sailor approached him and
asked his permission to speak.
The sailor told Sir Shovell that he was keeping track of their position by using his own navigational equipment. He also informed the admiral that the officers were badly mistaken and that the fleet was much closer to English shores than they anticipated. Therefore, according to the sailor it was very dangerous to proceed to England at full speed since they were in danger of wrecking on the Scilly Isles.
Sir Shovell ordered the man …hanged as a mutineer. Hours later, his flagship and three other boats of his fleet smashed into the rocks. He was swept ashore where, according to one of the legends, he was murdered by a woman who wanted his emerald ring.
By the way, the sailor has indeed, according to British Naval Law, deserved to be hanged. The danger of mutiny was a real problem on English ships due to bad living conditions, diseases and corporal punishment. Therefore only the officers were allowed to keep and use navigational equipment. Admiralty’s logic was that if the sailors didn’t know where they were, they would be less inclined to rebel against the officers.
This story was frequently used by historians to demonstrate the incompetence, stubbornness and, even pompousness, of some military leaders. Nonetheless, I sometimes joke with my students that the first mistake the admiral made was that of wasting his project resources. After all, what project manager in the right mind disposes of his precious project team members in the middle of the project?
On a more serious note, in my opinion this story also serves as a great example of how the project can go terribly wrong if the project manager chooses to ignore the critique and the feedback from the stakeholders in general and from his own “technical” team members in particular.
Introduction
I have already mentioned in “Defining The Detailed Scope: How And Where Do You Find Requirements?” the interesting paradox with walkthroughs, inspections and peer reviews. I have learned about these techniques fairly early in my career, liked the concept and started applying it on all of my projects fairly consistently and at times covertly, because of the pressure from some of my superiors to “quit wasting technical people’s time on ‘silly’ meetings”. I also know that many of my peers in the IT and software development industries utilized such practices on a regular basis as was confirmed by them during our discussions at various conferences and other
professional project management events.
Having spent a big chunk of my career in the aforementioned high-tech sector I naturally assumed that our project management cousins in the fields where project management has taken much deeper roots – engineering, construction and government – are using this tool on a regular basis.
To my great surprise the words “peer reviews” and “walkthroughs” did not generate much interest or recognition/familiarity among my BCIT (British Columbia Institute of Technology) project management students. Over the course of my teaching career at the university I had people from construction, engineering, pharmaceutical, government, utilities, oil and gas, movie and even theatre sectors sign up for my course. Surprisingly enough, very few of them (excluding some of the IT and software people) ever confirmed that they were indeed using or even aware of this powerful technique.
The interesting aspect of this phenomenon is that almost all of them, after a prolonged cheerleading from my end, tried using walkthroughs and peer reviews to validate the scope of the projects, to identify otherwise unexpected risks and to decrease the amount of mistakes and the rework required. The results were to say the least staggering – all of those who tried this methodology reported great results including, but not limited to, projects finished before the deadlines, high quality of final product or service and, most importantly, highly satisfied customers!
Peer Reviews, Inspections And Walkthroughs
Why Conduct Peer Reviews And Walkthroughs?
Let us start our discussion on the reviews with a very interesting statistic originating from the IT and software industry. Unfortunately due to lack of reliable information from other sectors we have to rely on the data collected in the high-tech field and then construct generalized extrapolations to the rest of the world. So, without further ado, here is a very interesting statistic:
Forty five percent (45%!) of project costs industry-wide is rework
Let’s ponder this number for a while … If we choose to believe in the power of extrapolation to one degree or another, it looks like by eliminating the amount of rework completely we can decrease all of our project budgets in half. Durations would probably fall by a significant number as well.
However, upon further analysis this number does not look excessively abnormal if one remembers the “Cost of Mistake” chart discussed in detail in “Kick-Starting Your Projects – What Should You Know About Defining Scope?” and “Defining The Detailed Scope: How And Where Do You Find Requirements?” articles. In a nutshell, the study conducted by Barry Boehm (one of the leading thinkers in the field of software development) claims that a mistake that would cost you one dollar to fix at the Initiation stage of the project will typically balloon to a $40 -1,000 deficiency by the project closure.
So, what can happen without the walkthroughs, inspections and peer reviews? Here is a very typical scenario (a bit exaggerated for illustrational purposes) witnessed by the author on numerous occasions throughout his career:
Step 1: Project manager, sometimes with the help of other resources (e.g. business analyst in IT field or architect in the construction industry, mechanical engineer in product development, etc.) develops a project plan and a scope document (e.g. Statement of Work).Step 2: Technical project team members are still working on other “very important” projects and their functional managers frown upon “distracting them from their urgenttasks.Step 3: Customers and/or end users are not consulted about the final designs either because they are “fairly technical” or to avoid “annoying” questions, corrections and other “disruptions”.Step 4: Project manager presents the documents – project plan and statement of work - to technical people and possibly does a quick walkthrough during the project kick-off meeting.Step 5: Project team members discover a great deal of deficiencies, mistakes and risks in the documentation, but unfortunately major commitments with respect to time, budget and scope have already been madeStep 6: Finally, the project team pulls together and based on sheer will and long hours deliver a product that somewhat resembles the scope described in the documentation. In addition, despite all of the heroic efforts the project is late and over the budgetStep 7: Finally, customers walk in the doors and even after cursory inspection of the product find dozens and dozens of defects, omissions and mistakes …
The subsequent sections of this chapter will attempt to explain what and how should be done with respect to walkthroughs, inspections and peer reviews in order to avoid the depressing situation described earlier. Exhibit 1 provides the reader wit a high-level overview of the documentation
review process.
How To Conduct Walkthroughs, Inspections And Peer Reviews?
What Documents Should Be Reviewed?
Typically it is a good idea to review all of the major project, scope and other key documents on your projects. The more documents get reviewed fairly early in the process, the less are the chances of missing scope items, unforeseen risks and hidden constraints. Having said that, in real life assembling the entire group of project stakeholders or even stakeholder sub-groups is a difficult and time-consuming task that may not be encouraged at some organizations. Therefore, it seems quite logical to concentrate, at least on the most important of the project documents.
Primary document for the project team members and customers would definitely be the Project Plan and the Statement of Work (aka Scope Document, aka Requirements Document) sine they contain the key information crucial to project success. It is also very useful to at least briefly familiarize the project team with the Project Charter in order to inform the technical team members about the project itself, key scope items and constraints. However it is absolutely essential that a project manager conducts a full-scale walkthrough of the Project Charter with the customers and key stakeholders.
How To Organize Peer Review and Walkthrough Meetings?
The best way to conduct peer reviews and walkthroughs (especially on larger, more complicated projects) is to arrange for them at regular intervals during the document development stage. At least two or three meetings should be scheduled closer to the end of Project Charter, Project Plan and Scope Document development stages.
Please note that there is a serious chance that the functional managers may tell you that the technical people are too busy to attend such meetings. Sometimes other stakeholders could feign busyness and refuse or ignore your invitations to the walkthrough meetings. Experience tells us that the best way to convince these people is to share the “Cost of Mistake” chart with them and ask them whether they consider spending a couple of hours now in order to avoid spending several hundred to a thousand hours later a worthwhile investment.
It is a good idea to reserve two-hour block of time in a medium to large-size well-ventilated conference room several days in advance. You will need at least two hours because the documents – especially on larger projects – tend to be fairly long. Thus, even walking a group of, say, twenty or thirty people through a fifty-page document will probably take the entire two-hour meeting.
On the other hand, keep in mind that walkthroughs and reviews of project documents, especially the ones taking place earlier in the process are very vigorous and spirited affairs that could drain all of project manager’s energy and even sometimes hope. Thus, allocating more than two hours for the meeting could have some inherent dangers – people will get tired, start losing their collective focus and may even become a bit cranky!
Always remember, walkthroughs and peer reviews can sometimes be a very humbling experience indeed! One advice I give to my students and my fellow project managers is that a couple of hours of humbling experience is a very small price to pay for discovering the aforementioned “one-dollar” mistakes, errors and omissions early in the project.
Preparing for the meeting also involves sending the first draft of the document in question to all the meeting participants with an indication that everyone is expected to review the document with a critical eye and come prepared to voice his or her concerns, suggestions and improvements. Mentioning that a document is just a first draft that could change dramatically after the series of reviews and walkthroughs is especially important because sometimes – and this depends on a number of cultural factors, including but not limited to organizational and country-specific – people could be reluctant to voice their criticism on your work.
I remember a situation when our project had to outsource some of the development work to India and after working in North America for my entire professional life and thus being used to a fair share of peer critique and feedback, I assumed that no adjustments should be made when communicating to the developers in Bangalore. I remember how we sent a Requirements Specification (Scope Document) to them and arranged for a conference call. I spent about thirty minutes walking our colleague through a fifteen-page document and concluded by asking the typical inspection-type question:
Me: “So, guys, now that I gave an overview of the scope, do you see any problems, risks, issues? Anything that would prevent you from doing proper design and development?”Team Lead in India: “No, everything seems perfect!”Me: “Guys, there is no such thing as a perfect first draft of the Requirements Specification! Come on, there must be something”Team Lead in India: “We don’t see any problems”Me: “OK, take a couple of days and review it amongst yourselves; if you have any questions, do not hesitate to contact me. We can discuss this again”Team Lead in India: “Great! We shall call you if anything comes up”
Needless to say I never received any follow-up phone calls or e-mails. While remaining in a blissful state of denial, although all of my experience was telling me that something was very wrong, I continued to receive progress reports from India stating that they were “four days ahead of schedule”. Finally when the product was delivered and inspected by our local team, we discovered that end-product was so flawed, that we had to redo the entire development phase of the project from scratch. Interestingly enough, we did discover a lot of issues and deficiencies in the original scope document that most definitely contributed to the low quality of the final product.
When I mentioned this story to one of my senior colleagues, a person who, by the way, was born and raised in India, he told me, “You really expected them to criticize your work?! Critique of one’s superiors is not something that is welcomed with open hands over there … They probably saw the mistakes and deficiencies in the document right away but did not want to embarrass you by pointing out the mistakes you made. It is very likely that they tried to address these deficiencies on their own and failed miserably. You should have inspected and peer reviewed it here first”.
The next step is to conduct a quick document walkthrough either using the document itself or some kind of presentation software. It is good idea to plug your laptop into the overhead projector so that everyone can see what is being discussed and recorded. I typically opt for a Word document so that all of the comments, critique, feedback and follow-up items can be recorded right there. It is a good idea to let people word their comments by themselves rather than rephrase them in format acceptable to you. This way you can avoid future misunderstandings about the content of the feedback.
Whatever form you decide to choose – comments in the word processor document or remarks in your notebook – it is worthwhile to re-record all of the action items from these meetings into some kind of “Issues List” and disseminate it to all of the meeting participants and other project stakeholders. Personal experience shows that incorporating this “Issues List” into the meeting minutes that should be created and sent out to all the stakeholders anyways is an effective and time saving technique.
Running Peer Reviews And Inspections
Before getting into the peer review discussion I would like to point out that there are actually two very distinct types of peer reviews. One can call them “proper format” review and “correct technical context” review.
The “proper format” review is typically done by an experienced project manager external to the project. Accordingly this person is usually not familiar with the detailed technical scope of the project. Actually in my experience it has been even better if that person hasn’t heard about the project in question before!
The problem is that someone who has been involved in the project extensively and has participated in many of the project discussions is less likely to be as alert and vigilant when reviewing project documentation. In addition, he or she may subconsciously be subject to the group assumptions, either conscious or unconscious, dominating the project team. A colleague of mine actually came up with a very accurate description of these phenomena; he referred to them
as “soaping of the eyes”.
For instance the technical people on the project team may think that they have a very good idea what a “seamless integration” means. An experienced peer reviewer who hasn’t been burdened by team influence will flag this phrase as inappropriate right away.
Therefore the “proper format reviewer’s” mission is not to validate the technical scope of the project or schedule and budget estimates, or to ask questions like, “What problem you were trying to solve?” His job is to make sure that the document follows the best practices of project management.
The documentation guru’s primary responsibility would be to capture deficiencies and mistakes such as ambiguous language (see Exhibit 4), missing measurability, completeness, consistency, prioritization, proper constructs, etc. (see article “Defining The Detailed Scope: How And Where Do You Find Requirements?” for an in-depth discussion of quality project documentation).
Unfortunately these kinds of reviews can not answer questions like:
-
- Can this scope item be delivered at all?
- Can this scope item be delivered with the budget and timeline provided?
- Are there any technical inconsistencies in the scope?
And this is where “correct technical context” reviews or inspections come in handy. The technical team members probably would focus on the correctness of technical information, unidentified risks, overlooked resource requirements, etc. in the documents rather than the way in which documentation is written.
For example, a statement like:
“Build a large four-bedroom three-bathroom Tuscan-style home on a budget of one hundred thousand dollars by the end of the next month”
will probably generate different kinds of feedback from the “guru” and from the technical project team members. The documentation guru will almost certainly notice an inappropriate use of the word “large” and will ask the project manager to associate a measurable attribute with that particular term. As a result the word “large” could be replaced with statements like “5,000 square feet” or “three-level, 6,000 square feet”.
On the other hand, technical team members may have an idea (not necessarily a correct one) of what “large” means or they may just miss it because of the lack of training in documentation reviews. However, they will probably come back to you with a comment that buildings of that size can not be built on a budget of hundred thousand dollars and be completed in month and a half.
Who are the people you should invite to peer reviews and inspections? With “proper format” reviews it is fairly straightforward – it’s the project manager and the documentation guru, most likely one of the most experienced project leaders in the company who has been properly trained in the fine art of critical reading.
The technical document inspections will require the presence of the project manager, team leads of each functional group involved in the project, all technical team members, quality assurance people, subject matter experts in each of the functional areas and several key representatives from the customer side.
Running Customer Walkthroughs
For a stakeholder walkthrough you probably want to consider inviting your clients or customer (whichever is applicable) and if possible the end users of the product. Note that each one of these groups could be in turn subdivided into several clusters; some of these clusters could be primary and some of them could be secondary but both are important.
Also, it would be a good idea to invite key representatives from the technical team since some of the questions posed by the customers and users could only be answered by them. You also need technical team reps in order to explain and justify your estimates.
If you are working on a multi-departmental project at a fairly functional organization inviting other department heads could also be valuable because they could end up providing resources for your project.
Peer Review And Walkthrough Challenges
Some of the issues encountered by the project teams during the peer reviews, walkthroughs and inspections include large requirements documents, large inspection teams and geographical separation.
So, what are the tools at your disposal in battling the “walkthrough fatigue”? First, if dealing with larger documents, attempt scheduling several reviews in stages. This way you can reduce the duration of reviews and maintain the focus of the participants. Also, try not to exceed a two hour time limit on meetings. And one final suggestion: when booking a conference room, pick the largest and the best ventilated room possible. Nothing kills the thought process like the lack of oxygen!
Sometimes the review teams can get very large, especially on key project documents like project plan or project scope definition. Having fifty or more people in the room commenting and providing critique on your documentation can turn into chaos very quickly. Therefore, it could be useful to break the inspection teams into groups for initial inspections, and once the major issues discovered in these separate meetings have been analyzed and addressed properly, the final joint technical team and customer walkthrough involving all the project stakeholders can take place.
Geographical separation can also present certain challenges. Try using tools like video conferencing, teleconferencing or tracking and comments functions in word processing software.
What Kinds Of Questions Should You Expect (And Ask)?
So what kinds of questions can one expect from the walkthroughs with the customers and the technical team members? Exhibit 2 lists some of the questions the project manager should expect when engaging in the walkthroughs, inspections and peer reviews.
What To Watch Out For: Defect Checklists
So what are the thing all of the reviewers should be on the watch out for during walkthroughs, peer reviews and inspections? When reviewing scope and other project management documents the stakeholders should be asking themselves the following questions:
Organization and completeness of the documentation:
- Are all parts of the documents written at a consistent and appropriate level of detail?
- Does the scope definition document provide an adequate basis for design?
- Do we have priorities assigned to each scope item?
- Is there any information missing in the document? Are there any “TBDs” in the
- documents?
- Did we cover all possible alternatives, exceptions, risks and constraints?
Technical feasibility:
- Is every requirement in scope?
- Can all the scope items be implemented with all the known constraints?
Measurability:
- Do all of the scope items have appropriate measurability attributes associated with them?
Ambiguity:
- Do all the key statements in the document have only one possible meaning?
- Can they be misinterpreted by any of the stakeholders?
“Dangerous words”
- Exhibit 3 provides a list of potentially troublesome words and phrases that tend to frequently appear in project management documentation in basically all of the industries and types of organizations
About the Author
Jamal Moustafaev, MBA, PMP – president and founder of Thinktank Consulting is an internationally acclaimed expert and speaker in the areas of project/portfolio management, scope definition, process improvement and corporate training. Jamal Moustafaev has done work for private-sector companies and government organizations in Canada, US, Asia, Europe and Middle East. Read Jamal’s Blog @ www.thinktankconsulting.ca
- Please follow me on Twitter:
- Like our page on Facebook:
- Connect with me on LinkedIn:
- Subscribe to my RSS feed:
Jamal is an author of two very popular books: Delivering Exceptional Project Results: A Practical Guide to Project Selection, Scoping, Estimation and Management and Project Scope Management: A Practical Guide to Requirements for Engineering, Product, Construction, IT and Enterprise Projects.