Simulation & Debugging Channel

Simulation enables designers to test their system without incurring the full cost and risk of destroying a system when using it in an untested capacity. This series focuses on the challenges and opportunities available to designers to simulate their systems through logical, physical, and hybrid hardware-in-the-loop simulation techniques.

Debugging Stories: Development Tools

Monday, January 31st, 2011 by Robert Cravotta

Anyone that has developed a system has debugging stories. A number of those stories are captured in the responses to a Question-of-the-Week posed a while ago about your favorite debugging anecdote. While collecting the different stories together reveals some worthwhile lessons learned, reading through all of the stories can be time consuming and random as to the type of content in each story. This article, and future others like it, will attempt to consolidate a class of debugging stories together to ease access for you. The rest of this article will focus on the stories and lessons based around issues with the development tools.

In reading through the stories, I am reminded that I worked with a C cross compiler that did not generate the proper code for declaring and initializing float variables. The work around was to avoid initializing the float variable as part of the declaration. The initialization had to be performed as a distinct and separate assignment within the body code. Eventually, within a year of us finding the problem, the company that made the compiler fixed it, but I continued to maintain the code so as to keep the declaration and initialization separate. It felt safer to the whole development team to comment the initialization value with the declaration line and place all of the initialization code at the beginning of the code block.

Two stories identified how the debugger can misrepresent how the actual runtime code executes with and without the debugger in the system. Andrew Coombes shared a story about how the debugger inappropriately assumed when a block of code had the same CRC value as the previously loaded code that it was identical and skipped the process of loading the new code onto the target. The problem was exacerbated by the fact that the debugger did not calculate the CRC correctly. S.B. @ LI shared a story where the debugger was intercepting and correcting the data types in a call structure to an operating system call. This masked the real behavior of the system when the debugger was not active where the data types were not correct.

There were stories about compilers that would allocate data to inappropriate or unavailable memory resources. RSK @ LI shared how he had to use an inline-like function using preprocessor macros to reduce the call depth to avoid overflowing the hardware stack. E.P. @ LI’s story does not specify whether the compiler set the cache size, but the debugged code used a cache block that was one database block large and this inappropriate sizing caused the database application to run excessively slow. R.D @ LI recounts how a compiler was automatically selecting a 14-bit register to store a 16-bit address value, and how adding a NOP in front of the assignment cause the compiler to choose the correct register type to store the value.

I recall hearing many admonishments when I was a junior member of the staff to not turn on the compiler optimizations. I would hear stories about compiler optimizations that did not mix well with processor pipelines that did not include interlocks, and the horrible behaviors that would ensue. J.N. @ LI recounts an experience with a compiler optimization that scheduled some register writes just before a compare so that the system behaved incorrectly.

M.B. @ LI reminds us that even library code that has been used for long periods of time over many projects can include latent problems – especially for functions embedded within libraries, such as newlib in this case. L.W. @ LI’s story tells of when he found a NULL pointer access that had been within a seldom activated conditional with a library call.

I like J.N. @ LI‘s summary – “Different tools have different strengths, which is why you learn to use several and switch off when one isn’t finding the problem. And sometimes one tool gives you a hint that gets you closer, but it takes a different tool (or tools) to get you the rest of the way.”

Please let me know if you find this type of article useful. If so, I will try to do more on the topics that receive large numbers of responses that can be grouped into a smaller set of categories.

What is your favorite debugging anecdote?

Wednesday, December 1st, 2010 by Robert Cravotta

We all know stories about how something went wrong during a project and how we or someone else was able to make a leap of logic that enabled them to solve the problem. However, I think the stories that stick with us through the years are the ones that imparted a longer term insight that goes beyond the actual problem we were trying to solve at the time. For example, I have shared two such stories from my days as a junior member of the technical staff.

One story centers around solving an intermittent problem that ultimately would have been completely avoided if the development team had been using a more robust version control process. The other story involves uncovering an unexpected behavior in a vision sensor that was uncovered only because the two junior engineers that were working with the sensor were encouraged to think beyond the immediate task they were assigned to do.

More than twenty years later, these two stories still leave me with two key insights that I find valuable to pass on to other people. In the version control story, I learned that robustness is not just doing things correctly, but involves implementing processes and mechanisms to be able to automatically self-audit the system. Ronald Reagan’s saying “Trust but verify” is true on so many levels. In the valuing uncertainty story, I learned that providing an appropriate amount of wiggle room in work assignments is an essential ingredient to creating opportunities to grow your team member’s skills and experience while improving the performance of the team.

I suspect we all have analogous stories and that when we share them with each other, we scale the value of our lessons learned that much more quickly. Do you have a memorable debugging anecdote? What was the key insight(s) you got out of it? Is it a story that grows in value and is worth passing on to other members in the embedded community? I look forward to seeing your story.