The crush of product and technology demonstrations at CES is over. As an attendee of the show, the vast majority of the product demonstrations I saw seemed to perform as expected. The technology demonstrations on the other hand did not always fare quite so well – but then again – the technology demonstrations were prototypes of possibly useful ways to harness new ideas rather than fully developed and productized devices. Seeing all of these demonstrations at the show reminded me of the prototypes I worked on and some of the spectacular ways that things could go wrong. I suspect that sharing these stories with each other will pass around some valuable (and possibly expensive) lessons learned to the group here.
On the lighter side of the mishap scale, I still like the autonomous robots that Texas Instruments demonstrated last year at ESC San Jose. The demonstration consisted of four small, wheeled robots that would independently roam around the table top. When they bumped into each other, they would politely back away and zoom off in another direction. That is, except for one of the robots which appeared to be a bit pushy and bossy as it would push the other robots around longer before it would back away. In this case, the robots were all running the same software. The difference in behavior had to do with a different sensitivity of the pressure bar that told the robot that it had collided with something (a wall or another robot in this case).
I like this small scale example because it demonstrates that even identical devices can take on significantly different observable behaviors because of small variances in the components that make up the device. It also demonstrates the possible value for closed-loop control systems to be able to access independent or outside reference points so as to be able to calibrate their behavior to some set of norms (how’s that for a follow-up idea on that particular demonstration). I personally would love to see an algorithm that allowed the robots to gradually influence each other’s behavior, but the robots might need more sensors to be able to do that in any meaningful way.
On the more profound side of the mishap scale, my most noteworthy stories involve live testing of fully autonomous vehicles that would maneuver in low orbit space. These tests were quite expensive to perform, so we did not have the luxury of “do-overs”. Especially in failure, we had to learn as much about the system as we could; the post mortem analysis could last months after the live test.
In one case, we had a system that was prepped (loaded with fuel) for a low orbit test; however, the family of transport vehicle we were using had experienced several catastrophic failures over the past year that resulted in the payloads being lost. We put the low-orbit payload system, with the fuel loaded, into storage for a few months while the company responsible for the transport vehicle went through a review and correction process. Eventually we got the go ahead to perform the test. The transport vehicle delivered its payload perfectly; however, when the payload system activated its fuel system, the seals for the fuel lines blew.
In this case, the prototype system had not been designed to store fuel for a period of months. The intended scenario was to load the vehicle with fuel and launch it within days – not months. During the period of time in storage, the corrosive fuel and oxidizer weakened the seals so that they blew when the full pressure of the fuel system was placed upon them during flight. A key takeaway from this experience was to understand the full range of operating and non-operating scenarios that the system might be subjected to – including being subjected to extended storage. In this case, a solution to the problem would be implemented as additional steps and conditions in the test procedures.
My favorite profound failure involves a similar low-orbit vehicle that we designed to succeed when presented with a three-sigma (99.7%) scenario. In this test though, there was a cascade of failures during the delivery to low-orbit phase of the test, which presented us with a nine-sigma scenario. Despite the series of mishaps leading to deployment of the vehicle, the vehicle was almost able to completely compensate for its bad placement – except that it ran out of fuel as it was issuing the final engine commands to put it into the correct location. To the casual observer, the test was an utter failure, but to the people working on that project, the test demonstrated a system that was more robust than we ever thought it could be.
Do you have any demonstration or testing mishaps that you can share? What did you learn from it and how did you change things so that the mishap would not occur again?
Tags: Failsafe
Processor Tracker
All Entries RSS Feed