Entries Tagged ‘Vision Sensing’

The battle for multi-touch

Tuesday, April 12th, 2011 by Robert Cravotta

As with most technologies used in the consumer space, they take a number of years to gestate before they mature enough and gain visibility to end users. Capacitive-based multi-touch technology burst into the consumer conscience with the introduction of the iPhone. Dozens of companies have since entered the market to provide capacitive touch technologies to support new designs and applications. The capabilities that capacitive touch technology can support, such as improved touch sensing for multiple touches, detecting and filtering unintended touches (such as palm and cheek detection), as well as supporting a stylus, continues to evolve and improve.

Capacitive touch enjoys a very strong position providing the technology for multi-touch applications; however, there are other technologies that are or will likely be vying for a larger piece of the multi-touch pie. A potential contender is the vision-based multi-touch technology found in the Microsoft Surface. However, at the moment of this writing, Microsoft has indicated that it is not focusing its effort for the pixel sense technology toward the embedded market, so it may be a few years before the technology is available to embedded developers.

The $7600 price tag for the newest Surface system may imply that the sensing technology is too expensive for embedded systems, but it is important to realize that this price point supports a usage scenario that vastly differs from a single user device. First, the Surface provides a 40 inch diagonal touch/display surface that four people can easily access and use simultaneously. Additionally, the software and processing resources contained within the system are sized to handle 50 simultaneous touch points. Shrink both of these capabilities down to a single user device and the pixel sense technology may become quite price competitive.

Vision-based multi-touch works with more than a user’s fingers; it can also detect, recognize and interact with mundane, everyday objects, such as pens, cups, paint brushes, as well as touch interface specific objects such as styli. The technology is capable, if you provide enough compute capability, to distinguish and handle touches and hovering of fingers and objects over the touch surface differently.I’m betting as the manufacturing process for the pixel sense sensors matures, the lower price points will make a compelling case for focusing development support to the embedded community.

Resistive touch technology is another multi-touch contender. It has been the work horse for touch applications for decades, but its ability (or until recently, lack of) to support multi-touch designs has been one of its significant shortcomings. One advantage that resistive touch has enjoyed over capacitive touch for single-touch applications is a lower cost point to incorporate it into a design. Over the last year or so, resistive touch has evolved to be able to support multi-touch designs by using more compute processing in the sensor to resolve the ghosting issues in earlier resistive touch implementations.

Similar to vision-based multi-touch, resistive touch is able to detect contact with any normal object because resistive touch relies on a mechanical interface. Being able to detect contact with any object provides an advantage over capacitive touch because capacitive touch sensing can only detect objects, such as a human finger or a special conductive-tipped stylus, with conductive properties that can draw current from the capacitive field when placed on or over the touch surface. Capacitive touch technology also continues to evolve, and support for thin, passive styli (with an embedded conductive tip) is improving.

Each technology offers different strengths and capabilities; however, the underlying software and embedded processors in each approach must be able to perform analogous functions in order to effectively support multi-touch applications. A necessary capability is the ability to distinguish between explicit and unintended touches. This capability requires the sensor processor and software to be able to continuously track many simultaneous touches and assign a context to each one of them. The ability to track multiple explicit touches relative to each other is necessary to be able to recognize both single- and multi-touch gestures, such as swipes and pinches. Recognizing unintended touches involves properly ignoring when the user places their palm or cheek over the touch area as well as touching the edges of the touch surface with their fingers that are gripping the device.

A differentiating capability for touch sensing is minimizing the latency between when the user makes an explicit touch and when the system responds or provides the appropriate feedback to that touch. Longer latencies can affect the user’s experience in two detrimental ways in that the collected data for the touch or gesture has poor quality or the delay in feedback confuses the user. One strategy to minimize latency is to sample or process less points when a touch (or touches) is moving; however, this risks losing track of small movements that can materially affect analyzing the user’s movement such as when writing their signature. Another strategy to accommodate tracking the movement of a touch without losing data is to allow a delay in displaying the results of the tracking. If the delay is too long though, the user may try to compensate and try to restart their movement – potentially confusing or further delaying the touch sensing algorithms. Providing more compute processing helps in both of these cases, but it also increases the cost and energy draw of the system.

While at CES, I experienced multi-touch with all three of these technologies. I was already familiar with the capabilities of capacitive touch. The overall capabilities and responsiveness of the more expensive vision-based system met my expectations; I expect the price point for vision-based sensing to continue its precipitous fall into the embedded space within the next few years. I had no experience with resistive-based multi-touch until the show. I was impressed by the demonstrations that I saw from SMK and Stantum. The Stantum demonstration was on a prototype module, and I did not even know the SMK module was a resistive based system until the rep told me. The pressure needed to activate a touch felt the same as using a capacitive touch system (however, I am not an everyday user of capacitive touch devices). As these technologies continue to increasingly overlap in their ability to detect and appropriately ignore multiple touches within a meaningful time period, their converging price points promise an interesting battle as each technology finds its place in the multi-touch market.

Touch with the Microsoft Surface 2.0

Tuesday, March 29th, 2011 by Robert Cravotta

The new Microsoft Surface 2.0 will become available to the public later this year. The technology has undergone significant changes from when the first version was introduced in 2007. The most obvious change is that the dimensions of the newer unitis much thinner, so much so, that the 4 inch thick display can be wall mounted – effectively enabling the display to act like a large-screen 1080p television with touch capability.Not only is the new display thinner, but the list price has nearly halved to $7600. While the current production versions of the Surface are impractical for embedded developers, the sensing technology is quite different from other touch technologies and may represent another approach to user touch interfaces that will compete with other forms of touch technology.

Namely, the touch sensing in the Surface is not really based on sensing touch directly – rather, it is based on using IR (infrared) sensors the visually sense what is happening around the touch surface. This enables the system to sense and be able to interact with nearly any real world object, not just conductive surfaces such as with capacitive touch sensing or physical pressure such as with resistive touch sensing. For example, there are sample applications of the Surface working with a real paint brush (without paint on it). The system is able to identify objects with identification markings, in the form of a pattern of small bumps, to track those objects and infer additional information about them that other touch sensing technologies currently cannot do.

This exploded view of the Microsoft Surface illustrate the various layers that make up the display and sensing housing of the end units. The PixelSense technology is embedded into the LCD layers of the display.

The vision sensing technology is called PixelSense Technology, and it is able to sense the outlines of objects that are near the touch surface and distinguish when they are touching the surface. Note: I would include a link to the PixelSense Technology at Microsoft, but it is not available at this time. The PixelSense Technology embedded in the Samsung SUR40 for Microsoft Surface replaces the five (5) infrared cameras that the earlier version relies on. The SUR40 for Microsoft Surface is the result of a collaborative development effort between Samsung and Microsoft. Combining the Samsung SUR40 with Microsoft’s Surface Software enables the entire display to act as a single aggregate of pixel-based sensors that are tightly integrated with the display circuitry. This shift to an integrated sensor enables the finished casing to be substantially thinner than previous versions of the Surface.

The figure highlights the different layers that make up the display and sensing technology. The layers are analogous to any LCD display except that the PixelSense sensors are embedded in the LCD layer and do not affect the original display quality. The optical sheets include material characteristics to increase the viewing angle and to enhance the IR light transmissivity. PixelSense relies on the IR light generated at the backlight layer to detect reflections from objects above and on the protection layer. The sensors are located below the protection layer.

The Surface software targets an embedded AMD Athlon II X2 Dual-Core Processor operating at 2.9GHz and paired with an AMD Radeon HD 6700M Series GPU using DirectX 11 acting as the vision processor. Applications use an API (application program interface) to access the algorithms contained within the embedded vision processor. In the demonstration that I saw of the Surface, the system fed the IR sensing to a display where I could see my hand and objects above the protection layer. The difference between an object hovering over and touching the protection layer is quite obvious. The sensor and embedded vision software are able to detect and track more than 50 simultaneous touches. Supporting the large number of touches is necessary because the use case for the Surface is to have multiple people issuing gestures to the touch system at the same time.

This technology offers exciting capabilities for end-user applications, but it is currently not appropriate, nor available, for general embedded designs. However, as the technology continues to evolve, the price should continue to drop and the vision algorithms should mature so that they can operate more efficiently with less compute performance required of the vision processors (most likely due to specialized hardware accelerators for vision processing). The ability to be able to recognize and work with real world objects is a compelling capability that the current touch technologies lack and may never acquire. While the Microsoft person I spoke with says the company is not looking at bringing this technology to applications outside the fully integrated Surface package, I believe the technology will become more compelling for embedded applications sooner rather than later. At that point, experience with vision processing (different from image processing) will become a valuable skillset.