Watching the BP oil pipe-capping live-video the other day made me think about formal procedures. Using a formal procedure is a way to implement the fault tolerant principle in cases where the operator is an integral part of the system’s decision process. Using a formal procedure helps to free up the operator’s attention while performing “routine tasks” so they can better recognize and react to the shifting conditions of a complex environment. Even though I could not see all of the video feeds or hear any of the audio feeds, I could imagine what was going on in each of the feeds during the capping operation.
I have worked on a fair share of path-finding projects that demonstrated the feasibility of many design ideas to make a completely autonomous vehicle that could complete complex tasks in real world environments. The design team would subject new ideas to substantive analysis and review before building a lab implementation that we could test. With each test-case success, we would move closer to testing the system in the target environment (space-based in many cases). One thing that impressed me then and has forever stayed with me is the importance and value of a formal operational procedure document.
I have written and reviewed many operational procedures. When I first started working with these procedures, I thought they were a bother that took a lot of time and effort to produce. We had to explicitly account for and document every little detail that we could imagine, especially the unlikely scenarios that we needed to detect during the system checkouts. We performed what seemed like endless walkthroughs of the procedures – each time refining small details in the procedure as different members of the team would share a concern about this or that. We did the walkthroughs so many times that we could complete them in our sleep. When I finally participated in my first live test, the value of those procedures and all of those walkthroughs became apparent – and they no longer seemed like a waste of time.
The formal procedure was an effective way to capture the knowledge of the many people, each with a different set of skills and specific knowledge about the system, in a single place. By performing all of those walkthroughs, it forced us to consider what we should do under different failure conditions without the burden of having to come up with a solution in real-time. The formal procedure enabled everyone on the team to be able to quickly perform complex and coordinated tasks that would be practically impossible to execute in an impromptu fashion – especially under stressful conditions.
The first and foremost reason for the procedures was to protect the people working around the system. We were dealing with dangerous materials where injuries or deaths were very real possibilities if we were not careful. The second reason for the procedures was to protect the system from operating in a predictable destructive scenario. Unpredictable failures are a common enough occurrence when you are working on the leading (bleeding?) edge because you are working in the realm of the unknown. The systems we were building only existed as a single vehicle or a set of two vehicles, and having to rebuild them would represent a huge setback.
The capping operation of the BP oil-pipe appears to encompass at least as much, if not significantly more, complexity than the projects I worked on so long ago. The video feed showed the ROV (remotely operated vehicle) robot arm disconnecting a blue cable from the capping structure. Then the video feed showed what I interpreted as the ROV operator checking out various points in the system before moving on to the next cable or step in some checklist. I could imagine the numerous go/no-go callouts from each of the relevant team members that preceded each task performed by the ROV operator. I am guessing that the containment team went through a similar process that we went through in building their formal procedures – first in the conference room, then in simulation, and finally on the real thing 5000 feet under the surface of the ocean.
While building and testing prototypes of your embedded systems may not involve the same adrenaline pumping excitement as these two scenarios, the cost of destroying your prototype system can be devastating. If you have experience using formal procedures while building embedded systems, and you would like to contribute your knowledge and experience to this series, please contact me at Embedded Insights.
[Editor's Note: This was originally posted on the Embedded Master]
Processor Tracker
All Entries RSS Feed
Take a look at the 2005 BP refinery explosion, an even more egregious case of “formal” procedures being ignored.
By the way, formal methods means something slightly different to a mathematically trained programmer. Formal methods in the mathematical sense also help reduce the error rates.
There is a danger in instituting heavyweight process, in that the formalities can substitute for substance if the team’s support for them begins to break down. Checking off the right boxes becomes a substitute for thinking and the error rate goes right back up.