Comments on: How long should testing changes take? http://www.embeddedinsights.com/channels/2011/12/21/how-long-should-testing-changes-take/ Shedding Light on the Hidden World of Embedded Systems Mon, 28 Jul 2014 16:18:37 -0400 hourly 1 http://wordpress.org/?v=3.0 By: Lisa Simone http://www.embeddedinsights.com/channels/2011/12/21/how-long-should-testing-changes-take/#comment-11204 Lisa Simone Wed, 28 Dec 2011 12:41:47 +0000 http://www.embeddedinsights.com/channels/?p=670#comment-11204 The degree of testing required depends on the change, as you alluded to, but it's more dependent upon the corresponding requirements and specifications. "Minimizing variations" between successive builds is a good idea in general, but many times "simple" changes are the cause of big problems - partly because someone assumed a change was simple. The change itself may be simple, but the implications on the requirements (and the final device performance) may be huge. Take for example, medical devices. A simple change may be to expand the range of reportable patient values for a diagnostic test. The change may be straightforward/simple to verify. However, it's possible that validation of the change could require clinical trials to ensure that it doesn't affect safety and effectiveness of the device. Thus, a simple change could take a significant amount of time and money to validate before it can be released. One might argue that medical devices are a "special case" but safety-critical features exist on a host of other applications such as automotive, industrial, and telecom. Making a simple change to reduce cost of a critical component may require new certifications. Making ANY change to a user interface would require consideration of user interface testing, except for some very simple changes like a misspelling. Changing a screen color or font size may affect readability - many UI changes inadvertently lead to use error because the change may alter how the user interacts with the device. Human Factors testing is often sorely lacking. So, the answer is the famous "it depends" - and it certainly does, on the requirements. And unfortunately, the quality of requirements is generally less than optimal. Is two weeks possible? Yes. Could two weeks be used as a benchmark? No way. The degree of testing required depends on the change, as you alluded to, but it’s more dependent upon the corresponding requirements and specifications. “Minimizing variations” between successive builds is a good idea in general, but many times “simple” changes are the cause of big problems – partly because someone assumed a change was simple. The change itself may be simple, but the implications on the requirements (and the final device performance) may be huge.

Take for example, medical devices. A simple change may be to expand the range of reportable patient values for a diagnostic test. The change may be straightforward/simple to verify. However, it’s possible that validation of the change could require clinical trials to ensure that it doesn’t affect safety and effectiveness of the device. Thus, a simple change could take a significant amount of time and money to validate before it can be released.

One might argue that medical devices are a “special case” but safety-critical features exist on a host of other applications such as automotive, industrial, and telecom. Making a simple change to reduce cost of a critical component may require new certifications. Making ANY change to a user interface would require consideration of user interface testing, except for some very simple changes like a misspelling. Changing a screen color or font size may affect readability – many UI changes inadvertently lead to use error because the change may alter how the user interacts with the device. Human Factors testing is often sorely lacking.

So, the answer is the famous “it depends” – and it certainly does, on the requirements. And unfortunately, the quality of requirements is generally less than optimal. Is two weeks possible? Yes. Could two weeks be used as a benchmark? No way.

]]>