Articles by Robert Cravotta

As a former Technical Editor covering Embedded Processing at EDN, Robert has been following and commenting on the embedded processing space since 2001 (see article index). His expertise includes software development and system design using microprocessors, microcontrollers, digital signal processors (DSPs), multiprocessor architectures, processor fabrics, coprocessors, and accelerators, plus embedded cores in FPGAs, SOCs, and ASICs. Robert's embedded engineering background includes 16 years as a Member of the Technical Staff at Boeing and Rockwell International working on path-finding avionics, power and laser control systems, autonomous vehicles, and vision sensing systems.

Have you experienced a “bad luck” test failure?

Wednesday, December 7th, 2011 by Robert Cravotta

Despite all of the precautions that the Mythbusters team takes when doing their tests, the team accidentally launched a cannonball into a neighborhood and through a home. The test consisted of firing a 6-inch cannonball out of a homemade cannon to measure the cannonball’s velocity. The cannonball was fired at a sheriff’s bomb disposal range, and it was supposed to hit large containers filled with water. The projectile missed the containers and made an unlucky bounce off a safety beam sending it into the nearby neighborhood. Luckily, despite the property damage, including careening through a house with people sleeping in it, no one was hurt.

This event reminds me of a number of bad luck test failures I have experienced. Two different events involved similar autonomous vehicle tests, but the failures were due to interactions with other groups. In the first case, we experienced a bad luck failure during a test flight that failed because we had delayed the test to ensure that the test could complete successfully. In this test, we had a small autonomous vehicle powered with rocket engines. The rocket fuel (MMH and NTO) is very dangerous to work with, so we handled it as little as possible. We had fueled up the system for a test flight when the word came down that the launch was going to be delayed because we were using the same kind of launch vehicle that had just experienced three failed flights before our test.

While we waited for the failure analysis to complete, our test vehicle was placed into storage with the fuel (there really was no way to empty the fuel tanks as the single-test system had not been designed for that). A few months later we got the go ahead on the test, and we pulled the vehicle out of storage. The ground and flight checkouts passed with flying colors and the launch proceeded. However, during the test, once our vehicle blew its ordnance to allow the fuel to flow through the propulsion system, the seals catastrophically failed and the fuel immediately vented. The failure occurred because the seals were not designed to be in constant contact with the fuel for the months that it was in storage. The good news was that all of the electronics were operating correctly, just that the vehicle had no fuel to do what it was intended to do.

The other bad luck failure was the result of poor communication about an interface change. In this case, the system had been built around a 100Hz control cycle. A group new to the project decided to change the inertial measurement unit so that it operated at 400Hz. The change in sample rate was not communicated to the entire team and the resulting test flight was a spectacular spinning out of control failed flight.

In most of the bad luck failures I am aware of, the failure occurred because of assumptions that masked or hid the consequences of miscommunication or unexpected decisions made by one group within the entire team. In our case, the tests were part of a series of tests and they mostly cost us precious time, but sometimes such failures are more serious. For example, the Mars Climate Orbiter (in 1999) unexpectedly disintegrated while executing a navigation command. The root cause of that failure/error was a mismatch in the measurement systems used. One team used English units while another team used Metric units.

I guess calling these bad luck failures is a nice way to say a group of people did not perform all of the checks they should have before starting their tests. Have you ever experienced a “bad luck” failure? What was the root cause for the failure and could a change in procedures have prevented it?

How should embedded systems handle battery failures?

Wednesday, November 30th, 2011 by Robert Cravotta

Batteries – increasingly we cannot live without them. We use batteries in more devices than ever before, especially as the trend to make a mobile version of everything continues its relentless advance. However, the investigation and events surrounding the battery fires for the Chevy Volt is yet another reminder that every engineering decision involves tradeoffs. In this case, damaged batteries, especially large ones, can cause fires. However, this is not the first time we have seen damaged battery related issues – remember the exploding cell phone batteries from a few years ago? Well that problem has not been completely licked as there are still reports of exploding cell phones even today (in Brazil).

These incidents remind me of when I worked on a battery charger and controller system for an aircraft. We put a large amount of effort into ensuring that the fifty plus pound battery could not and would not explode no matter what type of failures it might endure. We had to develop a range of algorithms to constantly monitor each cell of the battery and appropriately respond if anything improper started to occur with any of them. One additional constraint on our responses though was that the battery had to deliver power when it was demanded by the system despite parts of the battery being damaged or failing.

Even though keeping the battery operating as well as it can under all conditions represents an extreme operating condition, I do not believe it is all that extreme a condition when you realize that automobiles and possibly even cell phones sometimes demand similar levels of operation. I recall discussing the exploding batteries a number of years ago, and one comment was that the exploding batteries was a system level design concern rather than just a battery manufacturing issue – in most of the exploding phones cases at that time, the explosions were the consequence of improperly charging the battery at an earlier time. Adding intelligence to the battery to reject a charging load that was out of some specification was a system-level method of minimizing the opportunity to damage the batteries via improper charging.

Given the wide range of applications that batteries are finding use in, what design guidelines do you think embedded systems should follow to provide the safest operation of batteries despite the innumerable ways that they can be damaged or fail? Is disabling the system appropriate?

Food for thought on disabling the system is how CFL (compact fluorescent lights) handle end-of-life conditions for the bulbs when too much of the mercury has migrated to the other end of the lighting tube – they purposefully burn out a fuse so that the controller board is unusable. While this simple approach avoids operating a CFL beyond its safe range, it has caused much concern among the user population as more and more people are scared by the burning components in their lamp.

How should embedded systems handle battery failures? Is there a one size fits all approach or even a tiered approach to handling different types of failures so that users can confidently use their devices without fear of explosions and fire while knowing when there is a problem with the battery system and getting it fixed before it becomes a major problem?

What embedded device/tool/technology are you thankful for?

Thursday, November 24th, 2011 by Robert Cravotta

In the spirit of the Thanksgiving holiday being celebrated in the United States this week, what device, tool, or technology are you thankful for making your life as an embedded developer better? This can be anything, not just the newest developments. The idea is to identify those things that made a material impact during your embedded career.

As an example, my own introspection on the topic yielded a somewhat surprising answer for me – I am thankful for the humble USB port. I remember when my mentor came to me one day with a spec sheet and asked me what I thought about this thing called the Universal Serial Bus specification. We talked about that specification on and off over the course of a year or so, but never did I imagine the larger extent of the impact of so small a change.

Not only did USB simplify and make connecting and powering devices simpler and more reliable such that everyday technophobes could successfully use those devices, but it simplified development boards and workbenches all around because more and more of the new development boards that included a USB port no longer required a separate source for system power. As a consumer, I have recognized that I have not crawled under my desk to plug in a transformer in years – and for that, I am thankful.

There are many things I am thankful for, and these opportunities for introspection are valuable because they increase the odds of recognizing the mundane and nearly invisible changes that have permeated our environment – just like the invisible embedded devices that populate our everyday lives. What embedded device, tool, and/or technology are you thankful for?

Do you use hardware-in-the-loop simulation?

Wednesday, November 16th, 2011 by Robert Cravotta

While working on some complex control systems for aerospace projects, I had the opportunity to build and use hardware-in-the-loop (HIL or HWIL) simulations. A HIL simulation is a platform where you can swap different portions of the system between simulation models and real hardware. The ability to mix simulated and real components provides a mechanism to test and characterize the behavior and interactions between components. This is especially valuable when building closed-loop control systems that will perform in conditions that you do not fully understand yet (due to a lack of experience with the operating scenario).

Building a HIL simulation is an extensive effort. The simulation must be able to not only emulate electrical signals for sensors and actuators, but it may also need to be able to provide predictable and repeatable physical conditions, such as moving the system around on six degrees of freedom based on real or simulated sensor or actuator outputs. As a result, HIL can be cost prohibitive for many projects; in fact, to date the only people I have met that have used HIL worked on aircraft, spacecraft, automotive, and high-end networking equipment.

I suspect though with the introduction of more sensors and/or actuators in consumer level products, that HIL concepts are being used in new types of projects. For example, tablet devices and smartphones increasingly are aware of gravity. To date, being able to detect gravity is being used to set the orientation on the display, but I have seen lab work where these same sensors are being used to detect deliberate motions made by the user, such as shaking, lowering, or raising  the device. At that point, HIL concepts provide a mechanism for developers to isolate and examine reality versus their assumptions about how sets of sensors and/or actuators interact under the variation that can occur under each of these use scenarios.

In my own experience, I have used HIL simulation to characterize and understand how to successfully use small rocket engines to move and hover a vehicle in the air. The HIL simulation allowed us to switch between real and simulated engines that moved the system. This kind of visibility was especially useful because operating the vehicle was dangerous and expensive. Another HIL simulation allowed us to work with the real camera sensor and physically simulate the motion that the camera would experience in a usage scenario. In each of these simulation setups, we were able to discover important discrepancies between our simulation models and how the real world behaved.

Are HIL simulation concepts moving into “simpler” designs? Are you using HIL simulation in your own projects? Is it sufficient to work with only real hardware, say in the case of a smartphone, or are you finding additional value in being able to simulate specific portions of the system on demand? Are you using HIL in a different way than described here? Is HIL too esoteric a topic for most development?

Do you receive too much email?

Wednesday, November 9th, 2011 by Robert Cravotta

During a recent conversation I heard someone share a “good” thing about emailed newsletters – it is easy to sort the email list and block delete all of them. This got me thinking about how much time I spend managing emails each day, and it got me wondering, does anyone/everyone else receive on the order of 100 emails a day too? Mind you, these are the business relevant emails versus the countless spam emails that several layers of spam filters intercept and dispose of for me.

Email allows me to work with dozens of people asynchronously throughout the week without having to spend time identifying a common time that we can work together. However, I receive way too many newsletters such that I do not have time to open them all. On the other hand, many of them are not worth opening most of the time, but the few times they are worth opening make it worthwhile to receive them. A good subject line goes a long way to signaling whether a particular issue of a newsletter might be worth the time to open and read it. And that identifies one of the problems of receiving a constant stream of information – a majority of it is not relevant to what you need at the moment you receive it.

On the other hand, I do not like block deleting these emails because used correctly, they can provide a sort-of customized and pre-filtered search database for when I need to research something. I think this works because I choose which newsletters to receive, and it is easy (usually) to stop receiving a newsletter. When I do a search on this informal database, I sometimes find a pointer in a newsletter that helps me find the material I am looking for from sources that generally have earned my trust as being reliable.

The downside or cost of having these newsletters to search is that too many show up in my mailbox each week. Automatic email filtering rules that moves newsletters into folders are helpful, but they usually take place as the emails arrive in my mailbox. I prefer to have the newsletters pop up in my inbox so that I can see the subjects in them before they are shunted into a newsletter folder. To date, I have not seen an email tool that will move emails into appropriate folders after they have been in the inbox for a day or week.

Do you receive too much email? Or is your email box not overflowing with information? Are you receiving so many newsletters that aim to consolidate information for you but end up flooding your mailbox with too much information that is not immediately relevant? What strategies do you use to manage the influx of newsletters so that they do not interfere or possibly hide important emails?

Interface Transitions and Spatial Clues

Tuesday, November 8th, 2011 by Robert Cravotta

Every time I upgrade any of my electronic devices, there is a real risk that something in the user interface will change. This is true of not just updating software but also when updating hardware. While someone who is responsible for the update decided the change in the interface is an improvement over the old interface, there is often a jolt as established users either need to adjust to the change or the system provides mechanisms that support the older interface. Following are some recent examples I have encountered.

A certain browser for desktop computers has been undergoing regular automagical updates – among the recent updates is a shuffling of the menu bar/button and how the tabs are displayed within the browser. Depending on who I talk to, people either love or hate the change. Typically it is not that the new or old interface is better but that the user must go through the process of remapping their mental map of where and how to perform the tasks they need to do. For example, a new menu tree structure breaks many of the learned paths, the spatial mapping so to speak, to access and issue specific commands. This can result in a user not being able to easily execute a common command (such as clear temporary files) without feeling like they have to search for a needle in a haystack because their spatial mapping for a command needs to be remapped.

Many programs provide update/help pages to help with this type of transition frustration, but sometimes the update cycle for the program is faster than the frequency that a user may use a specific command, and this can cause further confusion as the information the user needs is buried in an older help file. One strategy to accommodate users is to allow them to explicitly choose which menu structure or display layout they want. The unfortunate thing about this approach is that it is usually an all or nothing approach. The new feature may only be available under the new interface structure. Another more subtle strategy that some programs use to accommodate users is to quietly support the old keystrokes while displaying the newer interface structure. This approach can work well for users that memorized keyboard sequences, but it does not help those users that manually traversed the menus with the mouse. Additionally, these approaches do not really help with transitioning to the new interface; rather, they enable a user to put off the day of reckoning a little longer.

My recent experience with a new keyboard and mouse provides some examples of how these devices incorporate spatial clues to improve the experience of adapting to these devices.

The new keyboard expands the number of keys available. Despite providing a standard QWERTY layout, the relative location of the keys on the left and right edge of the layout was different relative to the edge of the keyboard. At first, this caused me to hit the wrong key when I was trying to press keys around the corners and edges of the keyboard layout – such as the ctrl and the ~ keys. With a little practice, I no longer hit the wrong keys. It helps that the keys on the left and right edge of the layout are different shapes and sizes from the rest of the alphanumeric keys. The difference in shape helps provide immediate feedback of where my hands are within the spatial context of the keyboard.

Additionally, the different sets of keys are grouped together so that the user’s fingers can feel a break between the groupings and the user is able to identify which grouping their hands are over without looking at the keyboard. While this is an obvious point, it is one that contemporary touch interfaces are not able to currently accommodate. The keyboard also includes a lighting feature for the keys that allows the user to specify a color for the keyboard. My first impression was that this was a silly luxury, but it has proven itself a useful capability because it makes it possible to immediately and unambiguously know what context mode the keyboard is in (via different color assignments) so that the programmable keys can take on different functions with each context.

The new mouse expands on the one or two button capability by supporting more than a dozen buttons. I have worked with mice with many buttons before, and because of that, the new mouse had to have at least two thumb buttons. The new mouse though does a superior job not just with button placement, but in providing spatial clues that I have never seen on a mouse before. Each button on the mouse actually has a slightly different shape, size, and/or angle that it touches the user’s fingers. It is possible to immediately and unambiguously know which button you are touching without looking at the mouse. There is a learning curve to know how each button feels, but the end result is that all of the buttons are usable with a very low chance of pressing unintended buttons.

In many of the aerospace interfaces that I worked on, we placed different kinds of cages around the buttons and switches so that the users could not accidently flip or press one. By grouping the buttons and switches and using different kinds of cages, we were able to help the user’s hands learn how performing a given function should feel. This provided a mechanism to help the user detect when they might be making an accidental and potentially catastrophic input. Providing this same level of robustness is generally not necessary for consumer and industrial applications, but providing some level of spatial clues, either via visual cues or physical variations, can greatly enhance the user’s learning curve when the interface changes and provide clues when the user is accidently hitting an unintended button.

What tips do you have for estimating/budgeting?

Wednesday, November 2nd, 2011 by Robert Cravotta

Many years ago, during a visit to my doctor, he pointed out to me that I had visited him around the same time each year for the past few years for roughly the same symptoms – which were all stress related. It was at that moment when it finally dawned on me how stressful year-end budgeting activities were on me. It was also the moment when I understood how to focus my energy to minimize the amount of stress that this time of the year had on me by approaching the year-end budgeting activities from a different perspective.

I do not recall who I heard the expression “water off a duck’s back” from, but it probably has been a life saver for getting me successfully through many stressful events, including year-end budgeting. The expression brings images of ducks popping up to the surface of the water after diving under the water to eat. Remarkably, the water all rolls off the duck’s back and they are dry immediately after surfacing. I had a friend who had hair like that, but the expression “water off Paul’s head” is not quite as visually effective.

The stress of needing to take an accurate assessment of my project or department’s current condition coupled with having to project and negotiate for those resources we would need to accomplish our goals for the next planning period was much easier to handle if I could imagine the stress falling off me. Equally important in handling the extra stress of this time of year was realizing which goals were real and which goals were what we called management BHAGs (big hairy-a** goals).

My management at the time thought it was a good idea to purposefully create goals that they knew probably could not be attained in the hope that we might complete a significant portion of them with far fewer resources than we might otherwise expect to need. I’m not convinced that the BHAG approach works if you overuse it. If you have too many of them, or they are just too large of a leap, there is a huge temptation by the team to just write off the goal and internally adopt a more realistic goal anyway.

Going over earlier budgeting proposals and comparing them to what actually happened proved to be a helpful exercise. First, it provides a loose baseline for the new budget proposal. Second, it can provide a mechanism for improving your budgeting accuracy because you might notice a pattern in your budget versus actuals. For example, are the proposed budgets even close to the actuals? Are they too high or too low? Do the budgets/actuals trend in any direction? My experience showed that our tend line was fairly constant year over year, but that allocating a portion of the budget to acquiring and updating tools each year was an important part of keeping that cost line from trending upward as project complexities increased.

Do you know any useful tips to share about how to be more effective at estimating projects and performing planning and budgeting activities? Does any type of tool, including spreadsheets, prove especially useful in tracking past, present, and future projects and actuals? What are the types of information you find most valuable in developing a budget and schedule? How important is having a specific person, skill set, or tool set available to making projections that you can meet?

What makes a conference worth attending?

Wednesday, October 26th, 2011 by Robert Cravotta

Travel and training budgets for engineers (and possibly every profession) have taken a beating over the past decade. Gone are the “grand-ole-days” of huge conferences for developers that would take up multiple halls and you felt like a salmon swimming upstream to spawn as you walked the exhibit areas. Instead, show and expo attendance for these conferences is decidedly lighter than it used to be. The change in attendance has been so profound that a few single-company developer conferences were delayed, cancelled, or transformed into smaller regional workshops a few years ago.

The hopeful news is that there is a sense that design activity is beginning to pick up again. Conferences targeting developers seem to be returning, attendance is up, but it seems that most of the attendance is by locals who can avoid most of the expenses of travel. I was having a conversation with an exhibitor at a recent conference; we were discussing whether there was something that the conference could offer that would justify a growing attendance that includes more attendees that had to travel to get there. The conference offered many announcements for new or upcoming components and tools that will help developers build their projects faster, cheaper, and better. There were numerous workshops and hands-on training offerings that were all well attended.

One of the most obvious things gone from today’s conferences from the past is the over-the-top parties at the end of the day. I can’t help but wonder – tongue-in-cheek – whether the excessive parties were a symptom of the huge attendance or a motivator for it. As developers, we are creating more complex systems in the same or shorter time frames, often with smaller design teams than a few years ago.

On a more serious observation, the time and complexity pressures on developers today probably raise the bar on what a conference will deliver to developers to justify spending their time travelling and attending the conference. What makes a conference worth attending? What features or events have you seen that you thought were particularly valuable? Is there something missing from today’s conferences that makes them less valuable than previously? Or maybe the contemporary conference is doing exactly what you need it to do, but there is no time to attend. Do you find the recordings of the proceedings worthwhile or perhaps even sufficient?

What advice would you give to an embedded newcomer?

Wednesday, October 19th, 2011 by Robert Cravotta

Developing embedded systems is different from developing end applications. For one, when done correctly, no one knows or cares about the embedded system when deciding whether to buy the end device. I remember having to adjust my own expectation and method of describing what I did for a living. Saying I worked on the control system for a part of a larger project, I learned to accept telling the slightly less accurate answer to people about what I worked on – “I work on the Space Shuttle, or the Space Station, or a particular aircraft.” I know I am not alone in this sentiment based on the responses from our first question of the week more than a year and a half ago – “You know you’re an embedded developer when …

In addition to adjusting to a different way of describing my job to others, I learned a number of important concepts about designing embedded systems – many of which apply well to other aspects of life. For example, one lesson I learned the hard way is to avoid the urge to rewrite portions of a system just so it will operate “better”. Instead, make the smallest change possible to get the system to do what it needs to do correctly.

The value of this advice is most apparent when you are working on projects with aggressive schedules. There are usually so many things that absolutely need to be changed and added to a design that routing your effort to redesign and rewrite a portion of the system that is invisible to the end user anyway is a recipe for unhappiness and stress. Not only are you taking precious time away from doing what needs to be done to the system, but you are also producing new sources of errors into the system by touching parts that are not part of the scope of work you are supposed to be working on. The expression “If it ain’t broke, don’t fix it” captures the essence of this advice even if it does not impart why it is usually a good idea to adhere to.

I suspect that each of us knows a handful of good advice we could pass onto embedded newcomers or mentees. What advice would you give to someone learning the ropes of engineering and/or embedded design? Do you have a simple or catchy phrase that captures the essence of your advice? Or do you have an example that illustrates a non-obvious value of the piece of advice?

Does only one person dominate your designs?

Thursday, October 13th, 2011 by Robert Cravotta

The recent death of Apple’s former CEO, Steve Jobs, has been marked by many articles about his life. The products that Apple has launched and supported over the years have greatly influenced the technology markets. Many people are asking if Apple can maintain its technology position without Steve. If the design process at Apple was completely dominated by Steve, then this is a real question; however, if the design process proceeded in a similar fashion in all of the places I have worked before, then Apple should be able to continue doing what it has been doing for the past decade with much success.

Leonard E. Read’s article “I, Pencil: My Family Tree as told to Leonard E. Read” points out why Apple should be able to continue to prosper. This excerpt highlights its profound claim:

There isn’t a single person in all these millions, including the president of the pencil company, who contributes more than a tiny, infinitesimal bit of know-how. From the standpoint of know-how the only difference between the miner of graphite in Ceylon and the logger in Oregon is in the type of know-how. Neither the miner nor the logger can be dispensed with, any more than can the chemist at the factory or the worker in the oil field—paraffin being a by-product of petroleum.

The article applies this level of complexity to a pencil, which is simpler in composition and design than any of Apple’s products. If Steve acted correctly as a CEO, he did not allow himself to be a single point of failure for the company. Other people with similar talents (even if they are manifest across several people instead of just one person), should already be identified and integrated into the design process.

A key function for any manager is to be able to identify at-risk talents, skills, and experience within their groups and to create an environment where losing any single person will not kill the group’s ability to complete its tasks. The group’s productivity may suffer, but the tasks can be correctly completed.

Does the management of any large and successful company really allow its future to rest on the shoulders of a single individual? Does only a single person within your group dominate the design process so thoroughly that if they were to “win the lottery” and suddenly disappear that your group would be in trouble? What are some of the strategies your group uses to ensure that the loss of a single person becomes such a large risk to the completion of that project? Do you have a formal or informal process for cross training your team members?