Comments on: Do you refactor embedded software?

By: Farah

Farah — Thu, 19 Apr 2012 12:38:13 +0000

I used the term “Migration” as my research revolves round the transition of architectures in embedded systems from even-triggered to time-triggered designs. This is about changing the underlying architecture from multiple interrupts enabled in the pre existing system to change to a single interrupt based system.

By: J.D. @ LI

J.D. @ LI — Sun, 01 Apr 2012 14:28:54 +0000

@Don:
As you said, it takes a keen mind to capture multidisciplinary contexts, model dependencies and effects, design and implement hardware and firmware to fit the visualized functions, and then analyze, find the design problems and better it in a continuous base.
What we are talking about is dedicated, focused engineers. People that continuously interest themselves in improving.
The process of improving and create better designs is part of what constitutes excellent engineering.
And the attitude of taking ownership of the design, in the sense that you are responsible, as an engineer, for the performance and safety of the system, counts when you are defending a seemingly more difficult decision of refactor “perfectly good code” or completing “expensive and time-consuming tests”. That attitude, when coupled with common sense, is very convincing to the customer and Management.

There are several methodologies and quality assurance tools that can be used to formalize this process, even very simple and effective as the color coding you described.
But the truth is that, when you are fortunate to make the process work, everybody in the team feels proud of the work. That is one hell of a good indicator. Non-technical people that are distant from the implementation may be proud of bad products, but not the engineers who can see the hardware and firmware details.

It is not only about refactoring also. If you have that attitude (continuous improvement), your current design will be at least as good as the last one, but probably better, in several aspects. You must be able to pick at least 5 aspects from your current work that you can improve. If not now, in the next one. But when you see those aspects, fix them in the current design.

Don’t let the response from the field drive this process, but rather, make the field to benefit from it.

- Jonny

By: D.P. @ LI

D.P. @ LI — Sat, 31 Mar 2012 18:43:40 +0000

@Jonny: Unfortunate that you also agree this is a common behavior. With defect fixes on a shpping product there is buy in up front, as it is currently negatively impacting “the company”, and as you say, test plans, test equipment and so forth already exist. The critical area testing is “generally followed” by majority of my customers, likely all of them if I thought about it. I generally don’t disclose too much about the RTOS, LLC processes, but since “us contractors” deal with all types of customers from those “skilled in the art” to those with literally no product development expertise we do the following with “limited success”. In the primary detailed design document, called the Device Profile, we include requirements, followed by test plans, followed by test results and/or references to other doucments containing these items when “large”. When we enter a requirement it is done in “black”. When we right a test plan (detailed) we change the requirement and test plan to red text. The test result, in Device Profile, will mininmally follow the test plan and minimally state PASS or FAIL. This text remains “red” until it “passes” at which time we turn the related material to “blue”. When asked the test status we merely reply;, “is the entire document blue?” There are some customers/industries that regularly turn them completely blue, which is the exception. Some customers the product ships and the document is black, or almost totally black, menaing “informal testing may have occurred” but no proof of testing. I don’t know about others experience, but I find that when doing test plans/testing/capturing test results that one’s brain must use “different neuron paths” or something, as I ALWAYS find what I would call a defect. Sometimes critical and sometimes not, but I seem to ALWAYS find stuff. It’s been my experience you have a more “gobal perspective” of interrelated product functionality during testing and a more “focussed view” when implementing and during testing I generally find “inter-dependency issues” mroe than anything else, and generally due to “evolution of requirements or refinement of requirements” during implementation, where you often “miss crossing the tees and dotting all the i’s”. Since I obviously have a “black, red and blue” overall view of project status, even without a “single clue” of the content of the Devcie Profile you get an immediate sense of the project status. Yep, I have lived in the world of “defect databases”, spreadsheets and all that “tracking” stuff, but I don’t know about others, but it never seemed to give me a good feeling relative to the coverage testing. Of course our definition fo a requirment is “if you can’t test to it and verity it is met, it is NOT a requirement”. Just wondering, how many “really test to requirements” and how many “test for conformance to all requirements”? Also, any suggestions on a “better way” to indicate test status that is actually indicates the level of test coverage? (By the way, your word processor, that everyone has on their computer, is the only “tool” needed when you do it with colors, versus using the defect reporting tools most of my customers have no access and thus won’t use.).

By: J.D. @ LI

J.D. @ LI — Sat, 31 Mar 2012 18:43:17 +0000

@Don:
Your question (I presume it is not rhetorical) is central to the theme.
I am directly impacted by this paradoxically possible scenario:
- we need to have bug-free products;
- the route to that is good engineering coupled to comprehensive testing;
- a bug is, by stipulation, a software defect that hit the field;
- if you never release a product, you have a bug-free product;

Of course, you *must* release in a regular basis, so the company makes money etc.
The danger of Management forcing an early release of a firmware that is not sufficiently tested can be very damaging, especially in industrial systems, where bugs can potentially generate high losses and liability. Nevertheless, that scenario you described is surprisingly common.

What I did a few years ago, in my current company, is to force a minimum test coverage for *any* firmware, and declare that *all* firmwares have the highest criticality, i.e., operation on mission-critical environments. The test scenarios vary, but are based on the aspects changed in the firmware for the current release, on top of the checklist and regression tests currently in place. We simply do not sign off a release candidate that did not undergo the “minimum period” of testing.

The approval of the version involves an exposition of the aspects and associated risks of failure, so Management partakes in the decision to release, and Engineering applies pressure not to release early.

This process resulted in a significant reduction of fielded bugs. We have two product lines that are based on the 8051, and for which the releases were driven by field bug detections. After this protocol, we started to do planned refactoring and preemptive bug fixing, and almost eliminated bug impacts in the field.

- Jonny

By: D.P. @ LI

D.P. @ LI — Fri, 30 Mar 2012 17:50:54 +0000

Ya, perhaps I was a little to emotional in my statement, but the issue remains relative to the consequences of shpping a bad product. I acutally thought about this some more last night and know of varios “ship it” scenarios where an individual, and there were quite a few, that would ask at times during a test phase; “But does anyone know of any identified defect right now?” If you ask the question often enough during testing there will be times when no identified defects exist, although testing is not complete. Then when there are no response, product is shipped! My guess is this is somewhat common as I have seen it a few times. These are the “cases” where some consequences should exist, as it is a kind of knowing negligence. Stuff goes wrong in products when an unforseen scenario occurs that no onw was “intuitive enough” to foresee and thus a defect is uncovered. This is what I refer to the ship it syndrome, and how your typical engineer (with a career) would confront this behavior? I have also seen projects where there is poor engineering and you can test forever and it may never ship, which the root problem here is different. So what advice do you give those that ask since CYA is not the issue, fixing the problem is? Anyone can identify the issue, how do you guys fix it? “Not my job man” or “above my pa grade” is not the answer I am looking for here.

By: F.D. @ LI

F.D. @ LI — Fri, 30 Mar 2012 17:50:38 +0000

So what manager do you guys know, whom was fired after a bad product was shipped? If there are no consequences, this behavior does not change. However, you probably know of some engineers whom paid the price since they “didi it wrong”. Shipping a bad product, if it meant career disaster, might impact “just ship it” decisions.
——-
Fred Brooks wasn’t fired – but he was one of our country’s greatest pioneers. Anyone who repeats his mistakes once acquainted with them, should see a shrink for shock therapy.

By: F.D. @ LI

F.D. @ LI — Fri, 30 Mar 2012 17:50:08 +0000

Product quality is a “risk assessment issue” where I have NEVER seen any theories of calculations on “this issue” for an engineer to place in front of the “ship it” decision makers.
————
Well, I certainly have.

http://www.amazon.com/Principles-Of-Software-Engineering-Management/dp/0201192462

By: D.P. @ LI

D.P. @ LI — Thu, 29 Mar 2012 15:24:05 +0000

@Jonny,Frank & Eric: I have been watching, and commeting, on this issue for some time. You guys are “saying it like it is” and just have some minor additions. First I agree with all of you with some tweaks. I provide licensed, high quality, reusable assets to my customers and then interconnect them to make products. I send a “bill” for the liceinsing and the “work”. Thus the “cost of quality” can be measured. I sometimes deal with customers with no electronic product development knowledges and thus I see “normal human behavior” since those whom haven’t designed complex hardware or software can’t even relate to what you do every day. In effect, a product ships when the customer does not want to spend any more testing money. Companies with experence (generally large with deep pociets) tend to “do it right”. I have worked for two MAJOR control companies and when some division has a “recall” the “cost of lack of quality” can also be measured. Product quality is a “risk assessment issue” where I have NEVER seen any theories of calculations on “this issue” for an engineer to place in front of the “ship it” decision makers. It is like congress and the deficit and the budget, most people will put their head in the sand, determine what their next day will look like if the ship it, and kick the can down the road. This is not true for those whom understand the risks and open issues in a product. So what manager do you guys know, whom was fired after a bad product was shipped? If there are no consequences, this behavior does not change. However, you probably know of some engineers whom paid the price since they “didi it wrong”. Shipping a bad product, if it meant career disaster, might impact “just ship it” decisions. The old “cause & effect” discussion.

By: E.J. @ LI

E.J. @ LI — Thu, 29 Mar 2012 15:23:44 +0000

I have to agree with Frank – refactoring imposes a risk and that has to be balanced so that the business risk is acceptable, i.e. it is not a universally good thing to do.

I tend to leave working legacy code alone however much I hate it and how ever much I think it could be improved – why do more work and increase risk when there is no need, why expose your business to risk for no gain.

If I am changing code I may refactor some of the code involved if it makes my task easier, if it reduces the risk of errors, if it improves my ability to maintain and support the software in future.

This risk based approach means – I change only what is absolutely essential if a release is small and most of the code has been tested and is stable.

I do low risk refactoring if the change to the software is small, and the risk is smaller enough to justify the improvement.

I do larger refactoring for major releases, to reduce my risk and reduce my effort now or in the future, or pave the way for changes I expect to receive later.

Sometimes what a customer wants is a significant change that requires refactoring, and also justifies a lot of testing in I may do a considerable amount of refactoring, and then usually with my customers full agreement. (That way they share some of the risk, but also understand some of the benefit to their product.)

Refactoring on a large scale for a minor enhancement release is foolish – and a risk I will not take. I am in business not just a techy.

I guess I am saying balance the effort and risk of refactoring against the benefit of the release it is being added to. Your customers do not want refactoring they just want new features, so this is a commercial and technical decision, that has to balance benefit and risk.

By: F.D. @ LI

F.D. @ LI — Thu, 29 Mar 2012 15:23:21 +0000

While I believe strongly that code “that works” should evolve carefully into more robust representations, I don’t want to minimize the real-world risk that any change to a complex system will impact the short-term bottom-line. This risk is compounded by simultaneous feature creep.

Software Zen:

The inability of Management to correctly allocate time for design of “non-profit” standard unit tests, system tests, and field tests, which themselves introduce a different class of failure, is legendary. I suspect that unexpected software refactoring horror stories leads non-practicing Software Management to conclude that software engineers are prima donnas.whose pride of ownership and OCD regarding “elegant” code must be quietly discouraged.

The difference in attitude concerning the quality of a “working” product can be experienced first-hand by the non-civil-engineering traveller as roads transition between the United States and Mexico. Software, not so much.

The psychological aspect of Software Engineering puts a premium on all aspects of interpersonal relationships, communication skills and good general mental health.

Though our argot is subtle, and slightly subversive, it’s universal. On the other hand, being profoundly wise doesn’t meet deadlines by itself.