Anyone that has developed a system has debugging stories. A number of those stories are captured in the responses to a Question-of-the-Week posed a while ago about your favorite debugging anecdote. While collecting the different stories together reveals some worthwhile lessons learned, reading through all of the stories can be time consuming and random as to the type of content in each story. This article, and future others like it, will attempt to consolidate a class of debugging stories together to ease access for you. The rest of this article will focus on the stories and lessons based around issues with the development tools.

In reading through the stories, I am reminded that I worked with a C cross compiler that did not generate the proper code for declaring and initializing float variables. The work around was to avoid initializing the float variable as part of the declaration. The initialization had to be performed as a distinct and separate assignment within the body code. Eventually, within a year of us finding the problem, the company that made the compiler fixed it, but I continued to maintain the code so as to keep the declaration and initialization separate. It felt safer to the whole development team to comment the initialization value with the declaration line and place all of the initialization code at the beginning of the code block.

Two stories identified how the debugger can misrepresent how the actual runtime code executes with and without the debugger in the system. Andrew Coombes shared a story about how the debugger inappropriately assumed when a block of code had the same CRC value as the previously loaded code that it was identical and skipped the process of loading the new code onto the target. The problem was exacerbated by the fact that the debugger did not calculate the CRC correctly. S.B. @ LI shared a story where the debugger was intercepting and correcting the data types in a call structure to an operating system call. This masked the real behavior of the system when the debugger was not active where the data types were not correct.

There were stories about compilers that would allocate data to inappropriate or unavailable memory resources. RSK @ LI shared how he had to use an inline-like function using preprocessor macros to reduce the call depth to avoid overflowing the hardware stack. E.P. @ LI’s story does not specify whether the compiler set the cache size, but the debugged code used a cache block that was one database block large and this inappropriate sizing caused the database application to run excessively slow. R.D @ LI recounts how a compiler was automatically selecting a 14-bit register to store a 16-bit address value, and how adding a NOP in front of the assignment cause the compiler to choose the correct register type to store the value.

I recall hearing many admonishments when I was a junior member of the staff to not turn on the compiler optimizations. I would hear stories about compiler optimizations that did not mix well with processor pipelines that did not include interlocks, and the horrible behaviors that would ensue. J.N. @ LI recounts an experience with a compiler optimization that scheduled some register writes just before a compare so that the system behaved incorrectly.

M.B. @ LI reminds us that even library code that has been used for long periods of time over many projects can include latent problems – especially for functions embedded within libraries, such as newlib in this case. L.W. @ LI’s story tells of when he found a NULL pointer access that had been within a seldom activated conditional with a library call.

I like J.N. @ LI‘s summary – “Different tools have different strengths, which is why you learn to use several and switch off when one isn’t finding the problem. And sometimes one tool gives you a hint that gets you closer, but it takes a different tool (or tools) to get you the rest of the way.”

Please let me know if you find this type of article useful. If so, I will try to do more on the topics that receive large numbers of responses that can be grouped into a smaller set of categories.