Comments on: Question of the Week: Do you always use formal test procedures for your embedded designs?

By: G.f.W. @EM

G.f.W. @EM — Mon, 14 Jun 2010 20:53:45 +0000

You don't need an unlimited budget to develop and use formal test procedures, though it does require being able to manage your project development time wisely. Granted, if you're doing a one-off, cheaply mass-produced talking toy design and you'll never see the customer again after you scramble away with their money, perhaps formal test procedures aren't needed. Also, if you're a one-person shop with no hope or anticipation of ever expanding beyond that, perhaps they're not needed.

Everyone else should sit up and pay attention. Formal test procedures are, in the simplest terms, what you do to make sure the product works as expected. It ensures repeatability and reliability of your tests, reveals coverage gaps, and provides a map of your actions new team members. When there's more than one person on the team, the mantra should be document, document, document.

Also, I'm not sure what experience Anonymous had with BITE, but the analogy is a poor match. As a former military Avionics Technician, I had to use, troubleshoot, and repair dozens of different systems and more than a hundred different individual kinds of SRA (shop-repairable assemblies, or 'Black Boxes' to you and I). This encompassed everything from radio transmitter/receivers to emergency beacons to multi-processor multi-station airborne networked computer systems. Out of literally thousands of troubleshooting and repair evolutions I conducted in more than a decade of experience, I can recall 2 (yes, TWO) instances of BITE being the cause of a failure. However, there WERE multiple instances of troubleshooters not understanding how BITE was constrained in what it could test or what it was trying to tell them, and so assumed it had little worth. Sounds kind of like the misunderstanding we see with Formal Procedures, doesn't it?

Maybe that analogy holds up after all: You assume something's worthless if you don't actually understand the value it brings and how best to employ it.

By: R.S. @LI

R.S. @LI — Fri, 11 Jun 2010 17:04:59 +0000

Formal testing is factor that contributes critically to the quality. It needs completely different mindset as compared to development. Formal testing starts with understanding the product requirements/use cases clearly and developing test plans, procedures and test cases to ensure each of the requirements is tested. It also looks at which of the test cases can be automated and developing an automated test flow for them.

In my experience, I have seen that having an independent test team (this team does not develop product code) that is focused on testing the system (“breaking”) as a user gives a strong focus on quality of the overall deliverable.

The more test you do, the less bugs escape to the customer, less the cost of maintenance, better confidence on your deliverable and possibly repeat order from the customers. Formal testing is a business need – it is not a nice to have.

By: B.C. @LI

B.C. @LI — Thu, 10 Jun 2010 21:11:56 +0000

Yes, I worked at IBM, Kidde Aerospace, and now Itron. Each of these companies and the products were mission critical applications. There were large amounts of testing to support the design of these products. The quality managment process required formalized testing.
As formal test engineer working as a development engineer, I try to arrage my code to accomendate automated testing. Or place hooks within the code that will support testing. I think my designs are better by doing that.

By: Anonymous @EM

Anonymous @EM — Thu, 10 Jun 2010 21:07:35 +0000

Formal procedures are for academics and people with unlimited budgets. In 35 years of designing systems, not only have I never used or been asked to use formal procedures, I have rarely had time to do more than superficially test systems before they are shipped.

Formal procedures in some sense are analogous to the bloated “BITE” (Built-In-Test-Equipment) circuitry that the military included in control systems back in the 70s and 80s. Although on the surface it was a good idea, most times when the system shut down it was due to a problem in the BITE circuitry not the circuitry it was monitoring. Balance, as always, is the key to good engineering.