Eduardo’s response to first and second generation touch sensing provides a nice segue to clarifying third generation touch sensing capabilities. Eduardo said:

One other [classification] that I would say relates to “generations” is the single touch vs multitouch history; which I guess also relates [to] the evolution of algorithms and hardware to scan more electrodes and to interpolate the values between those electrodes. First generation: single touch and single touch matrixes; second generation: two touch, low resolution sliders; third generation: high resolution x-y sensing, multi touch detection.

While there is a “generational” shift between single- and multi-touch sensing, I am not sure the uses for multi-touch commands have reached a tipping point of adoption. My non-scientific survey of what types of multi-touch commands people know how to use yields only zoom and rotate commands. The MSDN library entry for touch provides an indication of the maturity of multi-touch interfaces; it identifies that Windows 7 supports new multi-touch gestures such as pan, zoom, rotate, two-finger tap, as well as press and tap. However, these multi-touch commands are more like manipulations where the “input corresponds directly to how the object would react naturally to the same action in the real world.”

I am excited about the possibilities of multi-touch interfaces, but I think standardizing gesture recognition for movements such as flicks, traces, and drags, which go beyond location and pen up/down data, is the relevant characteristic of third generation touch sensing. Nor is gesture recognition limited to touch interfaces. For example, there are initiatives and modules available that enable applications to recognize mouse gestures. The figures (from the referenced mouse gesture page) highlight a challenge of touch interfaces – how to provide feedback and a means for the user to visually see how to perform the touch command. The figure relies on an optional feature to display a “mouse trail” so that the reader can understand the motion of the gesture. The example figure illustrates a gesture command that combines tracing with a right-up-left gesture to signal to a browser application to open all hyperlinks that the trace crossed in separate tabs.

Open links in tabs (end with Right-Up-Left): Making any gesture ending with a straight Right-Up-Left movement opens all crossed links in tabs.

A common and useful mouse-based gesture that is not yet standard across touch sensing solutions is recognizing a hovering finger or pointer. Several capacitive touch solutions can technically sense a hovering finger, but the software to accomplish this type of detection is currently left to the device and application developer. An important component of detecting a hovering finger is detecting not just where the fingertip is but also what additional part of the display the rest of the finger or pointer is covering so that the application software can place the pop-up or context windows away from the user’s finger.

While some developers will invest the time and resources to add these types of capabilities to their designs today, gesture recognition will not reach a tipping-point until the software to recognize gestures, identify and filter out bad gestures, and abstract the gesture motion into a set of commands finds its way into IP libraries or operating system drivers.

I recently proposed a tipping point for technology for the third generation of a technology or product, and I observed that touch technology seems to be going through a similar pattern as more touch solutions are integrating third generation capabilities. It is useful to understand the difference between the different generations of touch sensing to better understand the impact of the emerging third generation capabilities for developers.

First generation touch sensing relies on the embedded host processor to support, and the application software to understand how to configure, control, and read, the touch sensor. The application software is aware of and manages the details of the touch sensor drivers and analog to digital conversion of the sense circuits. A typical control flow to capture a touch event consists of the following steps:

1)  Activate the X driver(s)

2) Wait a predetermined amount of time for the X drivers to settle

3) Start the ADC measurement(s)

4) Wait for the ADC measurement to complete

5) Retrieve the ADC results

6) Activate the Y driver(s)

7) Wait a predetermined amount of time for the Y drivers to settle

8) Start the ADC measurement(s)

9) Wait for the ADC measurement to complete

10) Retrieve the ADC results

11) Decode and map the measured results to an X,Y coordinate

12) Apply any sensor specific filters

13) Apply calibration corrections

14) Use the result in the rest of the application code

Second generation touch sensing usually encapsulates this sequence of steps to activate the drivers, measure the sensing circuits, and applying the filters and calibration corrections into a touch event. Second generation solutions may also offload the sensor calibration function, although the application software may need to know when and how to initiate the calibrate function. A third generation solution may provide automatic calibration so that the application software does not need to know when or how to recalibrate the sensor because of changes in the operating environment (more in a later article).

A challenge for providing touch sensing solutions is striking a balance between meeting the needs of developers that want low- and high-levels of abstraction. For low-level design considerations, the developer needs an intimate knowledge of the hardware resources and access to the raw data to be able to build and use custom software functions that extend the capabilities of the touch sensor or even improve its signal to noise ratio. For developers using the touch sensor as a high-level device, the developer may be able to work through an API (application programming interface) to configure, as well as turn on and off, the touch sensor.

The second and third generation touch API typically includes high-level commands to enable and disable, calibrate, and read and write the configuration registers for the touch sensor as well as low-level commands to access the calibration information for the touch sensor. The details to configure the sensor and the driver for event reporting differ from device to device. Another important capability that second and third generation solutions may include is the ability to support various touch sensors and display shapes without requiring the developer to rework the application code. This is important because for many contemporary touch and display solutions, the developer must be separately aware of the display, touch sensing, and controller components because there are not many options for fully integrated touch and display systems. In short, we are still in the Wild West era of embedded touch sensing and display solutions.

In a recent conversation with Ken Maxwell, President of Blue Water Embedded, Ken mentioned several times how third-generation touch controllers are applying dedicated hardware resources to encapsulate and offload some of the processing necessary to deliver robust touch interfaces. We talked about his use of the term third-generation as he seemed not quite comfortable with using it. However, I believe it is the most appropriate term, is consistent with my observations about third generation technologies, and is the impetus for me doing this hands-on project with touch development kits in the first place.

While examining technology’s inflections, I have noticed that technological capability is only one part of an industry inflection based around that technology. The implementation must also: hide complexity from the developer and user; integrate subsystems to deliver lower costs, shrink schedules, and simplify learning curves; as well as pull together multi-domain components and knowledge into a single package. Two big examples of inflection points occurred around the third generation of the technology or product: Microsoft Windows and the Apple iPod.

Windows reached an inflection point at version 3.0 (five years after version 1.0 was released) when it simplified the management of the vast array of optional peripherals available for the desktop PC and hid much of the complexity of sharing data between programs. Users could already transfer data among applications, but they needed to use explicit translation programs and tolerate the loss of data from special features. Windows 3.0 hid the complexity of selecting those translation programs and provided a data-interchange format and mechanism that further improved users’ ability to share data among applications.

The third generation iPod reached an industry inflection point with the launch of the iTunes Music Store. The “world’s best and easiest to use ‘jukebox’ software” introduced a dramatically simpler user interface that needed little or no instruction to get started and introduced more people to digital music.

Touch interface controllers and development kits are at a similar third generation crossroads. First-generation software drivers for touch controllers required the target or host processor to drive the sensing circuits and perform the coordinate mapping. Second-generation touch controllers freed up some of the processing requirements of the target processor by including dedicated hardware resources to drive the sensors, and it abstracted the sensor data the target processor worked with to pen-up/down and coordinate location information. Second generation controllers still require significant processing resources to manage debounce processing as well as reject bad touches such as palm and face presses.

Third-generation touch controllers integrate even more dedicated hardware and software to offload more context processing from the target processor to handle debounce processing, reporting finger or pen flicking inputs, correctly resolving multi-touch inputs, and rejecting bad touches from palm, grip, and face presses. Depending on the sensor technology, third-generation controllers are also going beyond the simple pen-up/down model by supporting hover or mouse-over emulation. The new typing method supported by Swype pushes the pen-up/down model yet another step further by combining multiple touch points within a single pen-up/down event.

Is it a coincidence that touch interfaces seem to be crossing an industry inflection point with the advent of third generation controllers, or is this a relationship that also manifests in other domains? Does your experience support this observation?

[Editor's Note: This was originally posted on Low-Power Design]

Embedded systems are, for the most part, invisible to the end user, yet the end application would not work properly without them. So what form does innovation take for something that is only indirectly visible to the end user? Swype’s touch screen-based text input method is such an example. The text entry method already exists on the Samsung Omnia II and releases on July 15 for the Motorola Droid X.

Swyping is a text entry method for use with touch screens where the user places their finger on the first letter of the word they wish to type. Without lifting their finger, the user traces their finger through each of the letters of the word, and they do not lift their finger from the touch screen until they reach the final letter of the word. For example, the figure shows the trace for the word quick. This type of text entry requires the embedded engine to understand a wider range of subtle motions of the user’s finger and couples that with a deeper understanding of words and language. The embedded engine needs to be able to infer and make educated guesses about the user’s intended word.

Inferring or predicting what word a user wishes to enter into the system is not new. The iPhone, as well as the IBM Simon (from 1993), use algorithms to predict and compensate for finger tap accuracy for which letter the user is likely to press next. However, swyping takes the predictive algorithm a step further and widens the text entry system’s ability to accommodate even more imprecision from the user’s finger position because each swyping motion, start to finish, is associated with a single word.

In essence, the embedded system is taking advantage of available processing cycles to extract more information from the input device, the touch screen in this case, and correlating it with a larger and more complex database of knowledge, a dictionary and knowledge of grammar in this case. This is analogous to innovations for other embedded control systems. For example, motor controllers are able to deliver better energy efficiency because they are able to collect more information about the system and environment they are monitoring and controlling. Motor controllers are measuring more inputs, inferred and directly, that allows them to understand not just the instantaneous condition of the motor, but also the environmental and operational trends, so that the controller can adjust the motor’s behavior more effectively and extract greater efficiency than earlier controllers could accomplish. They are able to correlate the additional environmental information with an increasing database of knowledge of how the system operates under different conditions and how to adjust for variations.

The Swype engine, as well as other engines like it, supports one more capability that is important; it can learn the user’s preferences and unique usage patterns and adjust to accommodate those. As embedded systems embody more intelligence, they move away from being systems that must know everything at design time and move closer to being systems that can learn and adjust for the unique idiosyncrasies of their operating environment.

[Editor's Note: This was originally posted on Low-Power Design]