How do you ensure full coverage for your design/spec reviews?

Wednesday, April 27th, 2011 by Robert Cravotta

Last week I asked whether design-by-committee is ever a good idea. This week’s question derives from my experience on one of those design-by-committee projects. In this particular project, I worked on a development specification. The approval list for the specification was several dozen names long – presumably the length of the approving signatures list should provide confidence that the review and approval process was robust and good. As part of the review and approval process, I personally obtained each signature on the original document and gained some insight into some of the reasoning behind each signature.

For example, when I approached person B for their signature, I had the distinct feeling they did not have time to read the document and that they were looking at it for the first time in its current form. Now I like to think I am a competent specification author, but this was a large document, and to date, I was the only person who seemed to be aware of the entire set of requirements within the document. Well, person B looked at the document, perused the signature list, and said that person E’s signature would ensure that the appropriate requirements were good enough for approval.

When I approached person D, they took two minutes and looked at two requirements that were appropriate to their skill set and responsibility and nothing else within the specification before signing the document. When it was person E’s turn at the document, I once again felt they had not had time to look at the document before I arrived for their signature. Person E looked at the signature list and commented that it was good that person B and D had signed off, so the document should be good enough for their signature. In this example, these three signatures encompassed only two of the requirements in a thick specification.

Throughout the review and approval process, it felt like no one besides me knew all of the contents of the document. I did good work on that document, but my experience indicates that even the most skilled engineers are susceptible to small errors that can switch the meaning of a requirement and that additional sets of eyes looking over the requirements will usually uncover them during a review. Additionally, the system-level implications of each requirement can only be assessed if a reviewer is aware of the other requirements that interact with each other. The design-by-committee approach, in this case, did not provide system-level accountability for the review and approval process.

Is this lack of full coverage during a review and approval cycle a problem unique to this project or does it happen on other projects that you are aware of? What process do you use to ensure that the review process provides appropriate and full coverage of the design and specification documents?


6 Responses to “How do you ensure full coverage for your design/spec reviews?”

  1. J.L. @ LI says:

    Be absolutely resolute and aggressive with a “You snooze, you lose” policy. Write up meeting notes with lists of attendees and invited absentees, topics, resolutions, to-dos, and do-by dates. Be hardheaded about refusing to allow changes after the meeting without forcing management buy-in of cost and schedule impact. Make people that ignore you pay a price of embarrassment and justification for it.

  2. A.T. @ LI says:

    Bringing a Louisville Slugger to meetings helps

  3. N.M. @ LI says:

    I don’t want to hurt anybody’s feelings but the quality of the answer has a lot to do with the quality of the question. If you give a group of people a large free text document and you say “Is this okay” you will not get any good answers. There has been some fascinating research on this phenomenon (as observed by Leonid, above): if you stage a mugging with one by-stander on the street, that person is likely to try to intervene but if you stage a mugging with lots of people on the street, it is likely that nobody will intervene because there is confusion about who, if anybody, is responsible.

    If you structure the specification into tables of requirements so that certain completeness properties can be easily checked (excellent research done on tabular requirements by David Gries, if I recall correctly) and you make each reviewer responsible for checking or approving specific aspects (and make it clear that their effort is not redundant and duplicated by other reviewers), you can expect to get MUCH better answers.

  4. L.R. @ LI says:

    This reminds me of a book I have recently been reading – “Freakonomics” – it tells a story about a group of people witnessing a crime, yet no one called the authorities – each one thought that the others have probably already did that, so they did not bother.
    When you divide responsibility among a large enough group, no one in particular feels responsible, and assumes the others have already checked most of the details.
    This is not to say that all team work is bad, just that one person must “own” a project and be completely accountable for its results, and in full authority of making decisions, to get something done well. The owner should still get critical advise from peers and subordinates, consider opposing views to his own, but have the guts to call the shots.

  5. D.G. @ LI says:

    Presumably, each reviewer in the long list had a reason to use the document in the future. Some users would be using the document as a basis for design. Some would be using the document to see if they had asked for the right product. No one is going to have the complete picture but they all should have an interest in certain content. Due to the lack of interest described, either the real users of the document were not on the list or no one intends to use the document. If the list was made up of managers then it is likely the real document users never saw it. If the list included actual document users then the downstream process does not incorporate the document enough for them to care. If the downstream process doesn’t incorporate the document then perhaps the process needs improvement. If the process is already optimized then the document has no purpose and should be eliminated.

    Utility, not coverage, is the goal. Supply side economics doesn’t work for documentation. Contracts often call out certain documents as deliverables. That encourages write-only document creation. What should be called out is evidence that certain documents were actually used. I say this now as if it is obvious but if it is so obvious why have I, and many others, create so many write-only documents?

  6. E.M. @ LI says:

    If there is an SOP that describes what the roles and responsibilities of the signing authorities, then there is not really much more a spec/design review should do. As far as traceability is concerned, generally requirements have a corresponding validation/verification effort that proves the design works as intended. The spec/design review is just a first step in the process that will catch some issues, but not all of them.

Leave a Reply