How long should testing changes take?

Wednesday, December 21st, 2011 by Robert Cravotta

The current two month payroll tax at the center of a budget bill going through the US Congress has elicited a response by a trade organization representing the people that would have to implement the new law and is the inspiration for this week’s question. A payroll processing trade organization has claimed that even if the bill became law, it is logistically impossible to make the changes in tax software before the two month extension expires. The trade organization claims the changes required by the bill would require at least 90 days for software testing alone in addition to time for analysis, design, coding and implementation. Somehow this scenario makes me think of past conversations where marketing/management would make changes to a system and engineering would push back because there was not enough time to properly perform the change and testing before the delivery date.

If you are part of the group requesting the “simple” change, you may think the developers are overstating the complexity of implementing the change. Often, in my experience, there is strong merit to the developer’s claims because the “simple” change involves some non-obvious complexity especially when the change affects multiple parts of the system.

In my own experience, we worked on many R&D projects, most with extremely aggressive time schedules and engineering budgets. These were quick and dirty proof of concepts many times and “simple” changes did not have to go through the rigorous production processes – or so the requesters felt. What saved the engineering team on many of these requests was the need to minimize the number of variations between iterations of the prototype so that we could perform useful analysis on the test data in the event of failures. Also, we locked down feature changes to the software during system integration so that all changes were in response to resolving system integration issues.

I suspect this perspective that changes can be made quickly and at low risk has been reinforced by the success of the electronics industry to deliver what appears to be the predictable and mundane advance in silicon products to cost 30% less and/or deliver twice as much performance as the previous year’s parts. Compounding this perception are all of the “science-based” television series that show complex engineering and scientific tasks being completed by one or two people in hours or days when in reality they would take dozens to hundreds of people months to complete.

How long should testing changes to a system take? Is it reasonable to expect any change to be ordered, analyzed, designed, implemented, and tested in less than two weeks? I realize that the length of time will depend on the complexity of the change request, but two weeks seems like an aggressive limit to implement any change which might indirectly affect the operation of the system. That is for embedded systems where the types of changes requested are much more complex than changing the color of a button or moving a message to a different portion of the display. How does your team manage change requests and the time it takes to process and implement them?

Tags: ,

One Response to “How long should testing changes take?”

  1. Lisa Simone says:

    The degree of testing required depends on the change, as you alluded to, but it’s more dependent upon the corresponding requirements and specifications. “Minimizing variations” between successive builds is a good idea in general, but many times “simple” changes are the cause of big problems – partly because someone assumed a change was simple. The change itself may be simple, but the implications on the requirements (and the final device performance) may be huge.

    Take for example, medical devices. A simple change may be to expand the range of reportable patient values for a diagnostic test. The change may be straightforward/simple to verify. However, it’s possible that validation of the change could require clinical trials to ensure that it doesn’t affect safety and effectiveness of the device. Thus, a simple change could take a significant amount of time and money to validate before it can be released.

    One might argue that medical devices are a “special case” but safety-critical features exist on a host of other applications such as automotive, industrial, and telecom. Making a simple change to reduce cost of a critical component may require new certifications. Making ANY change to a user interface would require consideration of user interface testing, except for some very simple changes like a misspelling. Changing a screen color or font size may affect readability – many UI changes inadvertently lead to use error because the change may alter how the user interacts with the device. Human Factors testing is often sorely lacking.

    So, the answer is the famous “it depends” – and it certainly does, on the requirements. And unfortunately, the quality of requirements is generally less than optimal. Is two weeks possible? Yes. Could two weeks be used as a benchmark? No way.

Leave a Reply