I commented earlier this week about how watching the BP oil well capping exercise live video reminded me about building and using formal test procedures when performing complex or dangerous operations. Before I had a lot of experience with test procedures, I used to think of them as an annoying check-off box for quality assurance. They were expensive to make, and they consumed huge amounts of time to build and refine. However, with more experience, I came to appreciate formal test procedures as valuable engineering design tools because they are a mechanism that injects fault tolerance in systems where the operator is an integral part of the system’s decision process. The procedure frees up the operator’s attention while performing “routine tasks” with the system so they can better recognize and react to the shifting external conditions of a complex environment.
Similar to the BP oil well capping exercise, the formal procedures I worked on involved complex systems that used dangerous chemicals. We needed to make sure we did not damage the systems while using them, both for safety and schedule reasons. Building the formal procedure and going through it with the entire team captured each member’s specialized knowledge so that the team was able to develop and refine each step in the procedure with a higher level of confidence than any subset of the team could have performed alone.
I personally understand the value of formal procedures for testing and operating very complex and dangerous systems, but I wonder if the formal procedure process offers similar value, compared to the cost and effort to make one, when applied to simple, low cost, or benign system designs.
Do you always build and use formal test procedures or are there designs that are so simple, low cost, or benign that you skip the process of building a formal test procedure? What types of designs would you consider skipping the formal procedure process and why?
If you would like to suggest questions for future posts, please contact me at Embedded Insights.
[Editor's Note: This was originally posted on the Embedded Master]
Processor Tracker
All Entries RSS Feed
Formal procedures are for academics and people with unlimited budgets. In 35 years of designing systems, not only have I never used or been asked to use formal procedures, I have rarely had time to do more than superficially test systems before they are shipped.
Formal procedures in some sense are analogous to the bloated “BITE” (Built-In-Test-Equipment) circuitry that the military included in control systems back in the 70s and 80s. Although on the surface it was a good idea, most times when the system shut down it was due to a problem in the BITE circuitry not the circuitry it was monitoring. Balance, as always, is the key to good engineering.
Yes, I worked at IBM, Kidde Aerospace, and now Itron. Each of these companies and the products were mission critical applications. There were large amounts of testing to support the design of these products. The quality managment process required formalized testing.
As formal test engineer working as a development engineer, I try to arrage my code to accomendate automated testing. Or place hooks within the code that will support testing. I think my designs are better by doing that.
Formal testing is factor that contributes critically to the quality. It needs completely different mindset as compared to development. Formal testing starts with understanding the product requirements/use cases clearly and developing test plans, procedures and test cases to ensure each of the requirements is tested. It also looks at which of the test cases can be automated and developing an automated test flow for them.
In my experience, I have seen that having an independent test team (this team does not develop product code) that is focused on testing the system (“breaking”) as a user gives a strong focus on quality of the overall deliverable.
The more test you do, the less bugs escape to the customer, less the cost of maintenance, better confidence on your deliverable and possibly repeat order from the customers. Formal testing is a business need – it is not a nice to have.
You don’t need an unlimited budget to develop and use formal test procedures, though it does require being able to manage your project development time wisely. Granted, if you’re doing a one-off, cheaply mass-produced talking toy design and you’ll never see the customer again after you scramble away with their money, perhaps formal test procedures aren’t needed. Also, if you’re a one-person shop with no hope or anticipation of ever expanding beyond that, perhaps they’re not needed.
Everyone else should sit up and pay attention. Formal test procedures are, in the simplest terms, what you do to make sure the product works as expected. It ensures repeatability and reliability of your tests, reveals coverage gaps, and provides a map of your actions new team members. When there’s more than one person on the team, the mantra should be document, document, document.
Also, I’m not sure what experience Anonymous had with BITE, but the analogy is a poor match. As a former military Avionics Technician, I had to use, troubleshoot, and repair dozens of different systems and more than a hundred different individual kinds of SRA (shop-repairable assemblies, or ‘Black Boxes’ to you and I). This encompassed everything from radio transmitter/receivers to emergency beacons to multi-processor multi-station airborne networked computer systems. Out of literally thousands of troubleshooting and repair evolutions I conducted in more than a decade of experience, I can recall 2 (yes, TWO) instances of BITE being the cause of a failure. However, there WERE multiple instances of troubleshooters not understanding how BITE was constrained in what it could test or what it was trying to tell them, and so assumed it had little worth. Sounds kind of like the misunderstanding we see with Formal Procedures, doesn’t it?
Maybe that analogy holds up after all: You assume something’s worthless if you don’t actually understand the value it brings and how best to employ it.