AT&T’s recent national outage of their U-Verse voice service affected me for most of one day last month. Until recently, such outages never affected me because I was still using a traditional landline phone service. That all changed a few months ago when I decided that the risk and consequences of an outage might be offset by the additional services and lower cost of the VoIP service over the landline service. Since the outage, I have been thinking about whether I properly evaluated the risks, costs, and benefits, and whether I should keep or change my services.

The impact of the outage was significant, but not as bad as it could have been. The outage did not affect my ability to receive short phone calls or send and receive emails. It did however severely reduce my ability to make outgoing phone calls and to maintain a long phone call as the calls that did get through would randomly drop. I had one scheduled phone meeting that I had to reschedule as a result of the outage. Overall, the severity and duration of the outage was not sufficient to cause me to drop the VoIP service in favor of the landline service. However, if more similar outages were to occur, say more frequently than on a twelve months cycle or for more than a few hours at a time, I might seriously reconsider this position.

An offsetting factor in this experience was my cell phone. My cell phone sort-of acts as my backup phone in emergencies, but it is insufficient for heavy duty activity in my office because I work at the edge of a wireless dead coverage spot in the mountains. I find it ironic that the cell phone has replaced my landline as my last line of defense to communicate in emergencies because I kept the landline so long as a last line of defense against the wireless phone service going down.

Many people are making this type of trade-off (knowingly or not). A May 12, 2010 report from the Centers for Disease Control and Prevention, says that 24.5% of American homes, in the last half of 2009, had only wireless phones. According to the repost, 48.6% of adults aged 25 to 29 years old lived in households with only wireless phones. The term VoIP never shows up in the report, so I cannot determine whether or not the data lumps landline and VoIP services into the same category.

Going with a wireless only household incurs additional exposures of single point of failure. 9-1-1 operators cannot automatically find you in an emergency. And in a crisis, such as severe storms,

the wireless phone infrastructure may overload and prevent you from receiving a cell signal.

The thing about single points of failure is that they are not always obvious until you are already experiencing the failure. Do you permit single point failures in the way you design your projects or in your personal life choices? For the purpose of this question, ignoring the possibility of a single point failure is an implied acceptance of the risk and benefit trade-off.

If you would like to suggest questions to explore, please contact me at Embedded Insights.

[Editor's Note: This was originally posted on the Embedded Master]

The recent articles claiming BP was demonstrating carelessness and complacency when the company cut corners in their well design got me thinking. Companies and design teams constantly improve their process in ways that cut costs and shrink schedules. This incremental process of “cutting corners” is the cornerstone of the amazing advances made in technology and improvements in our overall quality of life over the years. There seems to be a lot of second-guessing and criticism by people outside the design, build, and maintenance process when those incremental changes cause a system to cross the line between “good enough” and broken. The disaster that BP is in the middle of right now in the gulf is quite serious, but the magnitude of the disaster makes me think we should explore a thought exercise together.

What would be the cost/benefit if BP never “cut corners” on their well and rig designs? The immediately obvious answer might be, there would be no oil volcano at the bottom of the gulf right now and BP would be happily pumping oil out of that well instead of cleaning up after it. I say volcano because the words spill and leak seem so insufficient to describe the massive force required to spew out all of that oil despite the tremendous pressure pushing down on that opening by being under all of that water. A possible problem with the immediately obvious answer is that it ignores an essential implied assumption. While there might not be any oil pouring into the gulf, we might not be harvesting any of the oil either.

Let’s reword the question to make it more general. What would be the cost to society if everyone only engaged in ventures that would never find the line between good enough and broken? While raising my own children, I developed a sense of the importance that we all need to find the line between good enough and broken. I believe children do not break the rules merely to break the rules – I think they are exploring the edges and refining their own models of what rules are and why and when they should adhere to them. If we deny children the opportunity to understand the edges of rules, they might never develop the understanding necessary to know when to follow and when to challenge a rule.

This concept applies to engineering (as well as any human endeavor). If designers always use large margins in their designs, how will they know when and why they can or should not push those margins? How will they know if the margins are excessive (wasteful) or just right? My experience shows me that people learn the most from the failures, especially because it enables them to refine their models of how and why the world works the way it does.

I think one of the biggest challenges to “cutting corners” is minimizing the impact of when you cross the line to a failure precisely because you do not know where that line is. To me, derelict complacency depends on the assumption that the designer knew where the line to failure was and crossed it anyways. If my engineering career taught me anything, it taught me that we never know what will or will not work until we try it. We can extrapolate from experience, but experience does not provide certainty for everything we have not tried yet.

To an outsider, there might not be an easy to see difference between good engineering and derelict complacency. What are your thoughts on how to describe the difference between appropriate risk-assessed process improvement and derelict complacency? Can we use common failures in the lab to explore, refine, and communicate this difference so that we can apply it to larger disasters such as the oil in the gulf or even unintended acceleration in automobiles?

If you would like to suggest questions to explore, please contact me at Embedded Insights.

[Editor's Note: This was originally posted on the Embedded Master]

I commented earlier this week about how watching the BP oil well capping exercise live video reminded me about building and using formal test procedures when performing complex or dangerous operations. Before I had a lot of experience with test procedures, I used to think of them as an annoying check-off box for quality assurance. They were expensive to make, and they consumed huge amounts of time to build and refine. However, with more experience, I came to appreciate formal test procedures as valuable engineering design tools because they are a mechanism that injects fault tolerance in systems where the operator is an integral part of the system’s decision process. The procedure frees up the operator’s attention while performing “routine tasks” with the system so they can better recognize and react to the shifting external conditions of a complex environment.

Similar to the BP oil well capping exercise, the formal procedures I worked on involved complex systems that used dangerous chemicals. We needed to make sure we did not damage the systems while using them, both for safety and schedule reasons. Building the formal procedure and going through it with the entire team captured each member’s specialized knowledge so that the team was able to develop and refine each step in the procedure with a higher level of confidence than any subset of the team could have performed alone.

I personally understand the value of formal procedures for testing and operating very complex and dangerous systems, but I wonder if the formal procedure process offers similar value, compared to the cost and effort to make one, when applied to simple, low cost, or benign system designs.

Do you always build and use formal test procedures or are there designs that are so simple, low cost, or benign that you skip the process of building a formal test procedure? What types of designs would you consider skipping the formal procedure process and why?

If you would like to suggest questions for future posts, please contact me at Embedded Insights.

[Editor's Note: This was originally posted on the Embedded Master]