Requirements for tablet handwriting input

Monday, July 11th, 2011 by Guillaume Largillier

The market for tablet computers is exploding, with an estimated 100 million units that will be in use by 2013. The tablet form factor is very compelling because of its small size, light weight, and long battery life. Also, tablet operating systems, like Android, MeeGo, and iOs, have been designed so the input mechanism is touch-oriented, making their applications fun and easy to use.

One such tablet, the Apple iPad, has been a phenomenal success as a media-consumption device for watching movies, reading e-mail, surfing the Web, playing games, and other activities. But one thing is missing to make it the only device you need when going on a business trip: pen input. Pen input facilitates handwriting and drawing, which is important because our brain connectivity and the articulation in our arms, elbows, wrists, and fingers give us much greater motor skills when we use a tool. Everybody needs a pen, including children learning how to draw and write, engineers sketching a design, and business people annotating a contract.

Natural handwriting involves many contact points to be discriminated.

Accordingly, the ultimate goal for tablet designers should be to replicate the pen-and-paper experience. Pen and paper work so well together: resolution is high, there is no lag between the pen movement and the depositing of ink, and you can comfortably rest your palm on the paper, with its pleasing texture.

These qualities, however, are not easy to replicate in a tablet computer. You need high responsiveness, high resolution, good rendering, and palm rejection. Writing on a piece of glass with a plastic stylus is not an especially pleasing experience, so you also need coating on the glass and a shape and feel of a stylus that can approximate the pen and paper experience. Most importantly, you need an operating system and applications that have been designed from the ground up to integrate this input approach.

A key part of the equation is multi-touch technology that knows whether a particular finger touching the device is merely holding it or activating a menu, whether a certain contact point is a finger or a stylus, and, more importantly, it needs to distinguish a palm resting on the top of the display so a user can comfortably write on the device. In short, handwriting on a tablet computer doesn’t require multi-touch per se but smart detection of what’s happening on the screen.

The typical iVSM layout

Interpolated voltage sensing matrix (iVSM) is a multi-touch technology that provides smart detection, including handwriting rendering and palm rejection. It can allow users to simultaneously move a pen or a stylus (and an unlimited number of fingers) on a screen. iVSM is similar to projected capacitive technology, as it employs conductive tracks patterned on two superimposed substrates (made of glass or hard plastic). When the user touches the sensor, the top layer slightly bends, enabling electrical contact between the two patterned substrates at the precise contact location (Figure 3). A controller chip scans the whole matrix to detect such contacts, and will track them to deliver cursors to the host. However, whereas capacitive technology relies on proximity sensing, iVSM is force activated, enabling it to work with a pen, a stylus, or any number of implements. 

iVSM layout cut view

The tablet computer revolution is well underway around the world, with handwriting becoming an increasingly necessary function. Accordingly, device designers and vendors should take proper heed, or they might soon be seeing the handwriting on the wall.

Tags: ,
Posted in User Interfaces, Voices of Industry | 1 Comment »

What does the last Space Shuttle flight mean?

Wednesday, July 6th, 2011 by Robert Cravotta

The final Space Shuttle launch is scheduled for July 8, 2011. This upcoming event is a bittersweet moment for me and, I suspect, for many other people. I spent many years working in aerospace on projects that included supporting the Space Shuttle Main Engines as well as a payload that was cancelled for political (rather than technical) reasons after two years of pre-launch effort.

Similar to the tip of an iceberg, the Space Shuttle is just the front face of the launch and mission infrastructure that was the Space Shuttle program. Like many embedded systems that are contained within end systems, there is a huge amount of ground equipment and technical teams that work behind the scenes to make the Space Shuttle a successful endeavor. So one question is – what is the future of that infrastructure once the Space Shuttle program is completely closed down?

While the United States space program has been a largely publicly funded effort for many decades, the door is now opening for private entities to step up and take the stage. I am hopeful this type of shift will enable a resurgence in the space program because more ideas will be able to compete on how to best deliver space-based services rather than relying on a central group driving the vast majority of the direction that the space program could go. The flurry of aerospace activity and innovation that the Orteig Prize spawned demonstrated that private groups of individuals can accomplish Herculean feats – in this case, flying non-stop across the Pacific Ocean, in either direction, between New York and Paris.

However, I am not sure that a public prize is necessary to spawn a resurgence in aerospace innovation. There are a number of private space ventures already underway, including Virgin Galactic, SpaceX, as well as those companies in the list of private spaceflight companies on Wikipedia.

Does the end of the Space Shuttle program as it has been for the past few decades mean the space program will change? If so, how will it change – especially the hidden (or embedded) infrastructure? Is space just an academic exercise or are there any private/commercial ventures that you think will crack open the potential of space services that become self-sustaining in a private world?

Tags: , ,
Posted in Question of the Week, Random Insights | 2 Comments »

What game(s) do you recommend?

Thursday, June 30th, 2011 by Robert Cravotta

I have been thinking about how games and puzzles can help teach concepts and strengthen a person’s thought patterns for specific types of problem solving. However, there are literally thousands of games available across a multitude of forms, whether they are card, board, or computer-based games. The large number of options can make it challenging to even know when one might be particularly well suited to helping you train your mind for a type of design project. Discussion forums, like this one can collect lessons learned and make you aware of games or puzzles that others have found useful in exercising their minds – as well as being entertaining.

I have a handful of games that I could suggest, but I will start by offering only one recommendation in the hopes that other people will share their finds and thoughts about when and why the recommendation would be worthwhile to someone else.

For anyone that needs to do deep thinking while taking into account a wide range of conditions from a system perspective, I recommend checking out the ancient game of Go. It is a perfect knowledge game played between two players, and it has a ranking or handicap system that makes it possible for two players that are of slightly different strengths to play a challenging game for both players. Rather than explaining the specifics of the game here, I would instead like to focus on what the game forces you to do in order to play competently.

The rules are very simple – each player alternates turns placing a stone on a grid board. The goal of the game is to surround and capture the most territory. The grid is of sufficient size (19×19 points) that your moves have both a short term and a long term impact. Understanding the subtlety and depth of the long term impact of your moves grows in richness with experience and practice – not unlike designing a system in such a way as to avoid shooting yourself in the foot during troubleshooting. If you are too cautious, your opponent will capture too much of the board for your excellent long term planning to matter. If you play too aggressively – such as to capture as much territory as directly or as quickly as possible, you risk trying to defend what you have laid a claim to with a structure that is too weak to withstand any stress from your opponent.

The more I play Go, the easier I am able to see how the relationships between decisions and trade-offs affect how well the game – or a project – will turn out. Being able to find an adequate balance between building a strong structure and progressing forward at an appropriate pace is a constant exercise in being able to read your environment and adjusting to changing conditions.

I would recommend Go to anyone that needs to consider the system level impacts of their design decisions. Do you have a game you would recommend for embedded developers? If so, what is it and why might an embedded developer be interested in trying it out?

Tags:
Posted in Question of the Week, Robust Design | 8 Comments »

Touch me (too) tender

Tuesday, June 28th, 2011 by Robert Cravotta

A recent video of a fly activating commands on a touchscreen provides an excellent example of a touchscreen implementation that is too sensitive. In the video, you can see the computing system interpreting the fly’s movements as finger taps and drags. Several times the fly’s movement causes sections of text to be selected and another time you can see selected text that is targeted for a drag and drop command. Even when the fly just momentarily bounces off the touchscreen surface, the system detects and recognizes that brief contact as a touch command.

For obvious reasons, such over sensitivity in a touchscreen application is undesirable in most cases – that is unless the application is to detect and track the behavior of flies making contact with a surface. The idea that a fly could accidentally delete your important files or even send sensitive files to the wrong person (thanks to field auto-fill technology) is unpleasant at best.

Touchscreens have been available as an input device for decades, so why is the example of a fly issuing commands only surfacing now? First, the fly in the video is walking and landing on a capacitive touchscreen. Capacitive touch screens became more prevalent in consumer products after the launch of the Apple iPhone in 2007. Because capacitive touch screens rely on the conductive properties of the human finger, a touch command does not necessarily require a minimum amount of physical force to activate.

This contrasts with resistive touch screens which do require a minimum amount of physical force to cause two layers on the touch screen surface to make physical contact with each other. If the touch sensor in the video was a screen with a resistive touch sensor layered over it, the fly would most likely never be able to cause the two layers to make contact with each other by walking across the sensor surface; however, it might be able to make the surfaces contact each other if it forcefully collided into the screen area.

Touchscreens that are too sensitive are analogous to keyboards that do not implement an adequate debounce function for the keys. In other words, there are ways that capacitive touch sensors can mitigate spurious inputs such as flies landing on the sensor surface. There are two areas within the sensing system that a designer can work with to filter out unintended touches.

The first area to address in the system is to properly set the gain levels so that noise spikes and small conductive objects (like the feet and body of a fly) do not trigger a count threshold that would be interpreted as a touch. Another symptom of an oversensitive capacitive touch sensor is that it may classify a finger hovering over the touch surface as a touch before it makes contact with the surface. Many design specifications for touch systems explicitly state an acceptable distance above the touch surface that can be recognized as a touch (on the order of a fraction of a mm above the surface). I would share a template for specifying the sensitivity of a touch screen, but the sources I checked with consider that template proprietary information.

One reason why a touch system might be too sensitive is because the gain is set too high so as to allow the system to recognize when the user is using a stylus with a small conductive material within its tip. A stylus tip is sufficiently smaller than a human finger, and without the extra sensitivity in the touch sensor, a user will not be able to use a stylus because the sensor will fail to detect the stylus tip near the display surface. Another reason a touch system could be too sensitive is to accommodate a use-case that involves the user wearing gloves. In essence, the user’s finger never actually makes contact with the surface (the glove does), and the sensor system must be able to detect the finger through the glove even though it is hovering over the touch surface.

The other area of the system a designer should address to mitigate spurious and unintended touches is through shape processing. Capacitive touch sensing is similar to image or vision processing in that the raw data consists of a reading for each “pixel” in the touch area for each cycle or frame of input processing. In addition to looking for peaks or valleys in the pixel values, the shape processing can compare the shape of the pixels around the peak/valley to confirm that it is a shape and size that is consistent with what it expects. Shapes that are outside the expected set, such as six tiny spots that are close to each other in the shape of a fly’s body, can be flagged and ignored by the system.

This also suggests that the shape processing should be able to track context because it needs to be able to remember information between data frames and track the behavior of each blob of pixels to be able to recognize gestures such as pinch and swipe. This is the basis of cheek and palm rejection processing as well as ignoring a user’s fingers that are gripping the edge of the touch display for hand held devices.

One reason why a contemporary system, such as the one in the video, might not properly filter out touches from a fly is that the processor bandwidth of the processing used for the shape processing algorithm could not perform the more complex filtering in the time frame allotted. In addition to actually implementing additional code to handle more complex tracking and filtering, the system has to allocate enough processing resources to complete those tasks. As the number of touches that the controller can detect and track increases, the amount of processing required to resolve all of those touches goes up faster than linearly. Part of the additional complexity for complex shape processing comes from determining which blobs are associated with other blobs and which ones are independent from the others. This correlation function requires multi-frame tracking.

This video is a good reminder that what is good enough in the lab might be completely insufficient in the field.

Tags: , ,
Posted in User Interfaces | No Comments »

How is embedded debugging different?

Wednesday, June 22nd, 2011 by Robert Cravotta

Despite all the different embedded designs I worked on, one of the projects that stands out the most is the first embedded project I worked on – despite the fact that I already had ten years of experience with programming computers before that. I had received money for writing simulators, database engines, an assembler, a time share system, as well as several automation tools for production systems. All of these projects executed on mainframe systems or desktop computers. None of them quite prepared me for how different working on an embedded design is.

My first embedded design was a simple box that would reside on a ground equipment test rack that supported the flight system we were building and demonstrating. There was nothing particularly special about this box – it had a number of input and select lines and it had a few output lines. What surprised me most when putting it through its first checkout tests was how clueless I was as to how to troubleshoot the problems that did arise.

While I was aware of keyboard debounce routines from using my desktop system, I had never had to so completely understand the characteristics of different types of switches before. I had never before had to be aware of the wiring within the system, nor had I ever even considered doing an end-to-end check on every wire in a system ever before. While putting this simple box together, I became aware of so many new ways a design could go wrong that I had never had to consider in my earlier designs.

On top of the new ways that the system could behave incorrectly, the system had no file system, no display system, and no way to print out a trace log or memory dump. This made debugging a very different experience. Printf statements would be of no use, and there was no single-step debugger available. Worse yet, running the target program on my desktop computer, so that it could simulate the code, was mostly useless because I could not bring the real-world inputs and outputs that the box worked with into the desktop system.

As I tackled each debugging issue, I went from a befuddled state of having no idea how to proceed to a state where I adopted new ways of thinking that let me gain the insights I needed to infer how the system was (or was not) working and what needed to change. I worked on that project alone, and it welcomed me into the world of embedded design and working with real world signals with wide open arms.

How did your introduction to embedded systems go? What insights can you share to warn those that are entering the embedded design community about how designing, debugging, and integrating embedded components is different from writing application-level software?

Tags:
Posted in Question of the Week, Simulation & Debugging, Software Techniques | 6 Comments »

What techniques do you use to protect information?

Thursday, June 16th, 2011 by Robert Cravotta

The question of how to protect information on computers and networks has been receiving a lot of visibility with public disclosures of more networks being hacked over the past few weeks. The latest victims of hacking in the last week include the United States CIA site, the United States Senate site, and Citibank. Based on conversations with people about mine and their own experiences with having account and personal information compromised, I suspect there are a number of techniques that each of us use that could prove useful to share with each other on how to improve the protections on your data.

Two techniques that I have started to adopt in specific situations involve the use of secure tokens and the use of dummy email addresses. The secure token option is not available for every system, and it does add an extra layer of passwords to the login process. The secure token approach that I use generates a new temporary passcode every 30 seconds. Options for generating the temporary passcode include using a hardware key-fob or a software program that runs on your computer or even your cell phone. The secure token approach is far from transparent, and there is some cost in setting up the token.

I have only just started playing with the idea of using temporary or dummy email addresses to provide a level of indirection between my login information and my email account. In this case, my ISP allows me to create up to 500 temporary email ids that I can create, manage, and destroy at a moment’s notice. I can create a separate email address for each service. What makes these email addresses interesting though is that there is no way to actually log into the email account with those names as they are merely aliases for my real account which remains private. I’m not positive if this is a better way than just using a real email address, but I know I was worried the one time I had an service hacked because I realized that the email address that was connected to that service was also used by other services – and that represented a potential single point of failure or security access point to a host of private accounts.

One challenge of the dummy email accounts is keeping track of each one; however, because there is no access point available for any of these addresses, I feel more comfortable using a text file to track which email address goes to which service. On the other hand, I am careful to never place the actual email address that I use to access those dummy addresses in the same place.

Do you have some favorite techniques that you have adopted over the years to protect your data and information? Are they techniques that require relying on an intermediary – such as with the secure tokens, or are they personal and standalone like the dummy email address idea? Are any of your techniques usable in an embedded device, and if so, does the design need to include any special hardware or software resources to include it in the design?

Tags:
Posted in Question of the Week, Security & Encryption | 2 Comments »

Are GNU tools good enough for embedded designs?

Wednesday, June 8th, 2011 by Robert Cravotta

The negative responses to the question about Eclipse-based tools surprised me. It had been at least four years since I tried an Eclipse-based development tool, and I assumed that with so many embedded companies adopting the Eclipse IDE that the environment would have cleaned up nicely.

This got me wondering if GNU-based tools, especially compilers targeting embedded processors, fare better within the engineering community or not. Similar to using the Eclipse IDE, it has been far too many years since I used a GCC compiler to know how it has or has not evolved. Unlike an IDE, a compiler does not need to support a peppy graphical user interface – it just needs to generate strong code that works on the desired target. The competition to GCC compilers are proprietary tools that claim to perform significantly better at generating target code.

Are the GNU-based development tools good enough for embedded designs – especially those designs that do not provide a heavy user interface? The software for most embedded designs must operate within constrained memory sizes and need to operate efficiently or it will risk driving the cost of the embedded system higher than it needs to be.

Are you using GNU-based development tools – even when there is a proprietary compiler available for your target? What types of projects are GNU-based tools sufficient for and where is the line when the proprietary tools become a necessity (or not)?

Posted in Question of the Week, Simulation & Debugging | 4 Comments »

What is important when looking at a processor’s low power modes?

Wednesday, June 1st, 2011 by Robert Cravotta

Low power operation is an increasingly important capability of embedded processors, and many processors support multiple low power modes to enable developers to accomplish more with less energy. While low power modes differ from processor to processor, each mode enables a system to operate at a lower power level either by running the processor at lower clock rates and voltages or by removing power from selected parts of the processor, such as specific peripherals, the main processor core, and memory spaces.

An important characteristic of a low power or sleep mode is the current draw while the system is operating in that mode. However, evaluating and comparing the current draw between low power modes on different processors requires you to look at more than the just current draw to perform an apples-to-apples comparison. For example, the time it takes the system to wake-up from a given mode can disqualify a processor from consideration in a design. The time it takes a system to wake up is dependent on such factors as the settling time for the clock source and for the analog blocks. Some architectures offers multiple clock sources to allow a system to perform work at a slower rate while the faster clock source is still settling – further complicating the comparison between the wake-up time for the processor.

Another differentiator for low power modes is the level of granularity the power modes support that allows the developer to turn on and off individual versus blocks of peripherals or coprocessors. Some low power modes remove power from the main processor core and leave an autonomous peripheral controller operating to manage and perform data collection and storage. Low power modes can differ on which circuits they leave running such as brown-out detection, preserving the contents of ram or registers, and whether the real time clock remains active. The architectural decisions of which circuits can be powered down or not depends greatly on the end application, and they provide opportunities for specific processors to best target niche requirements.

When you are looking at a processor’s low power modes, what do you consider the important information that must be considered? When considering different processors, do you compare wake-up times or does current draw trump everything else? How important is your ability to control which circuits are powered on or off?

Tags:
Posted in Energy Management, Question of the Week | No Comments »

Do you care if your development tools are Eclipse based?

Wednesday, May 25th, 2011 by Robert Cravotta

I first explored the opportunity of using the Eclipse and Net Beans open source projects as a foundation for embedded software development tools in an article a few years back. Back then these Java-based IDEs (Integrated Development Environments) were squarely targeting application developers, but the embedded community was beginning to experiment with using these platforms for their own development tools. Since then, many companies have built and released Eclipse-based development tools – and a few have retained using their own IDE.

This week’s question is an attempt to start evaluating how theses open source development platforms are working out for embedded suppliers and developers. In a recent discussion with IAR Systems, I felt like the company’s recent announcement about an Eclipse plug-in for the Renesas RL78 was driven by developer request. IAR also supports its own proprietary IDE – the IAR Embedded WorkBench. Does a software development tools company supporting two different IDEs signal something about the open source platform?

In contrast, Microchip’s MPLAB X IDE is based on the Net Beans platform – effectively a competing open source platform to Eclipse. One capability that using the open source platform provides is that the IDE supports development on a variety of hosts running Linux, Mac OS, and Windows operating systems.

I personally have not tried using either an Eclipse or Net Beans tool in many years, so I do not know yet how well they have matured over the past few years. I do recall that managing installations was somewhat cumbersome, and I expect that is much better now. I also recall that the tools were a little slower to react to what I wanted to do, and again, today’s newer computers may have made that a non-issue. Lastly, the open source projects were not really built with the needs of embedded developers in mind, so the embedded tools that migrated to these platforms had to conform as best they could to architectural assumptions that were driven by the needs of application developers.

Do you care if an IDE is Eclipse or Net Beans based? Does the open source platform enable you to manage a wider variety of processor architectures from different suppliers in a meaningfully better way? Does it matter to your design-in decision if a processor is supported by one of these platforms? Are tools based on these open source platforms able to deliver the functionality and responsiveness you need for embedded development?

Tags: ,
Posted in Question of the Week, Software Techniques | 34 Comments »

An interface around the bend

Tuesday, May 24th, 2011 by Robert Cravotta

User interface options continue to grow in richness of expression as sensor and compute processing costs and energy requirements continue to drop. The “paper” computing device is one such example, and it hints that touch interfaces may only be the beginning of where user interfaces are headed. Flexible display technologies like E-Ink’s have supported visions of paper computers and hand held computing devices for over a decade. A paper recently released by Human Media Lab explores the opportunities and challenges of supporting user gestures that involve bending the device display similar to how you would bend a piece of paper. A video of the flexible prototype paper phone provides a quick overview of the project.

The paper phone prototype provides a demonstration platform for exploring gestures that involve the user bending the device to issue a command.

The demonstration device is based on a flexible display prototype called paper phone (see Figure). The 3.7” flexible electrophoretic display is coupled with a layer of five Flexpoint 2” bidirectional bend sensors that are sampled at 20Hz. An E-Ink Broadsheet AM 300 Kit with a Gumstix processor that is capable of completing a display refresh in 780ms for a typical full screen grey scale image. The prototype is connected to a laptop computer that offloads the processing for the sensor data, bend recognition, and sending images to the display to support testing the flexibility and mobility characteristics of the display.

The paper outlines how the study extends prior work with bend gestures in two important ways: 1) the display provided direct visual feedback to the user’s bend commands, and 2) the flexible electronics of the bending layer provided feedback. The study involved two parts. The first part asked the participants to define eight custom bend gesture pairs. Gestures were classified according to two characteristics based on the location of the force being exerted on the display and the polarity of that force. The configuration of the bend sensors supported recognizing folds or bends at the corners and along the center edge of the display. The user’s folds could exert force forward or backwards at each of these points. Gestures could consist of folding the display in a single location or at multiple locations. The paper acknowledges that there are other criteria they could have used, such as the amount of force in a fold, the number of repetitions of a fold, as well as tracking the velocity of a fold. These were not investigated in this study.

The second part of the study asked participants to use and evaluate the bend gestures they developed in the context of complete tasks, such as operating a music player or completing a phone call. The study found that there was strong agreement among participants for the folding locations as well as the polarity of the folds for actions with clear directionality, such as navigating left and right. The applications that the participants were asked to complete were navigating between twelve application icons; navigating a contact list; play, pause, and select a previous or next song; navigate a book reader, and zoom in and out for map navigation. The paper presents analysis of the 87 total bend gestures that the ten participants created (seven additional bends were created in the second part of the study) in building a bend gesture/language, and it discusses shared preferences among the participants.

A second paper from Human Media Lab presents a demonstration “Snaplet” prototype for a bend sensitive device to change its function and context based on its shape. The video of the Snaplet demonstrates the different contexts that the prototype can recognize and adjust to. Snaplet is similar to the paper phone prototype in that it uses bend sensors to classify the shape of the device. Rather than driving specific application commands with bends, deforming the shape of the device drives which applications the device will present to the user and what types of inputs it will accept and use. The prototype includes pressure sensors to detect touches, and it incorporates a Wacom flexible tablet to enable interaction with a stylus. Deforming the shape of the device is less dynamic than bending the device (such as in the first paper); rather the static or unchanging nature of the deformations allows the device’s shape to define what context it will work in.

When the Snaplet is bent in a convex shape, such as a wristband on the user’s arm, Snaplet acts like a watch or media player. The user can place the curved display on their arm and hold in in place with Velcro. The video of the Snaplet shows the user interacting with the device via a touch screen with icons and application data displayed appropriately for viewing on the wrist. By holding the device flat in the palm of their hand, the user signals to the device that it should act as a PDA. In this context, the user can use a Wacom stylus to take notes or sketch; this form factor is also good for reading a book. The user can signal the device to act as a cell phone by bending the edge of the display with their fingers and then placing the device to their ear.

Using static bends provides visual and tangible cues to the user of the device’s current context and application mode. Holding the device in a concave shape requires a continuous pinch from the user and provides haptic feedback that signifies there is an ongoing call. When the user releases their pinch of the device, the feedback haptic energy directly corresponds with dropping the call. This means that users can rely on the shape of the device to determine its operating context without visual feedback.

The paper phone and Snaplet prototypes are definitely not ready for production use, but they are good demonstration platforms to explore how and when using bend gestures and deforming the shape of a device may be practical. Note that these demonstration platforms for these types of inputs do not suggest replace existing forms of user interfaces, such as touch and stylus inputs; rather, they demonstrate how bend gestures can augment those input forms and provide a more natural and richer communication path with electronic devices.

Tags: , ,
Posted in User Interfaces | 1 Comment »

« Older Entries
Newer Entries »