Operation of a computer with touch screen interface

Apple’s latest patent application dealing with a touch sensitive tablet (PA# 20060053387). Inventor: Bas Ording.

It is desirable to use the information about a user’s actions to make the user experience with the computer, via the touch screen virtual keyboard, more powerful than is usual (or, perhaps, even possible) with computers using external keyboards.

Bas Ording has been thinking.

Mainly, he thinks about how to make aqua buttons, widgets, sheets and copy dialog boxes.

Here’s a nice quote from one of Bas’s patents that I think speaks volumes on the design philosophy at Apple:
With respect to manipulating windows (e.g., opening, closing, sizing, repositioning), conventional operating systems primarily focus on efficiency. However, operating systems do not focus on the aesthetics associated with these operations, particulary minimizing, maximizing and restoring operations. Accordingly, it would be desirable to provide more aesthetically pleasing operations, while continuing to provide all of the functionality associated with traditional techniques.

Here we get the benefit of his (further) thoughts about how a virtual keyboard ought to work. It’s not as simple as I would have thought (Okam’s razor has been thrown out the window) but simple is not the aim of the game in UI design- intuitive and consistent design and liberal use of the genie effect are the higher truths.

Bas hasn’t worked the genie effect into this patent (though he did invent it) but there are a few nice features. The essence of the patent is the use of the computer to analyse touch input on a virtual keyboard and determine the users intent. Is this just someone’s hand resting on the keyboard (i’d like that sort of feature of a physical keyboard when i place a book accidentally on the edge of the keyboard and produce a screen full of ZZZZZZZZZZZs). There are lots of interesting questions; What does the user expect from holding down a key for 100ms or 1sec?, What about different amounts of pressure in a tap?, How could a user signal that they made a mistake (via a gesture) and don’t want to type that key?, could a user change an e into an é by performing some action?

Touch screen keyboards, by nature of the hardware on which they operate, typically glean much more information about a user’s actions than can be gleaned with a typical external keyboards. For example, whereas a typical external keyboard includes a single “sensor” (such as a mechanical switch or electrical connection) or a small number of sensors per key, touch screen keyboards typically have many more sensors per virtual key.

Somewhat frustratingly, the patent application doesn’t answer any of the questions it poses. Essentially, the application says ‘an OS with a touch screen virtual key ought to take note of the duration of touch and pressure and then respond accordingly’- What the appropriate reaction ought to be isn’t really covered.

Preview of the key the user has hit for small keyboards

In some examples, a “preview” function is provided. Thus, for example, before a virtual GUI item’s activation is committed, an indication of the activation is displayed. This may be particularly useful, for example, where the virtual GUI items are particularly small. FIG. 11 illustrates a finger 1102 whose touch may be activating either the “w” virtual key or the “e” virtual key. The temporary overlay 1108 displays an indication that the “e” virtual key is to-be activated, before the “e” virtual key activation is committed as typed text 1110. For example, the finger 1102 being lifted while a particular virtual key is indicated as being to be activated may cause the particular virtual key to actually be activated. In the FIG. 11 example, if the “e” virtual key is not to be activated, then the finger 1102 may be moved (e.g., without lifting it) on the touch screen to activate a different virtual key.

Secondary actions based on gestures and GUI feedback

FIG. 9 illustrates a possible reaction is display of a menu 902. For example, the user may activate the “e” virtual key 906 with a particular behavior, and the reaction includes display of the menu 902. The menu 902, in the FIG. 9 illustration, includes additional virtual keys (e.g., 904a and 904b) corresponding to different forms of “e.” In some examples, actual display of the menu 902 is bypassed based on particular behaviors, although the virtual keys of the menu 902 are still present and able to be activated. For example, the user may nominally activate the “e” virtual key 906 with a “brush” gesture toward the additional virtual key 904 that is desired to be activated. The user would know the location of the additional virtual keys 904 based, for example, on previous use operation of the virtual keyboard. Not displaying the menu 902, a shortcut is provided for activating the additional virtual keys

More Complex interactions with buttons

FIG. 10 illustrates an example of a display having virtual GUI items that are not virtual keys of a virtual keyboard. The general concepts discussed above with the specific example of virtual keys of a virtual keyboard are generally applicable to virtual GUI items. For example, less pressure and/or shorter duration of touching (e.g., by a finger 1002) may be required to activate the “keep” virtual GUI item 1004 than to activate the “delete” virtual GUI item 1006. In addition, the processing may be such that activation of some GUI items (such as the “delete” GUI item, for example) require additional behavior, such as a “wiggle” or other gesture, which would serve as additional confirmation that it is intended for the GUI item to be activated.

An example of responses to touches of different duration

Another patent is still to come: “ACTIVATING VIRTUAL KEYS OF A TOUCH-SCREEN VIRTUAL KEYBOARD”

Looking to buy a new musical instruments? Zzounds has all sorts of musical instruments ranging from midi keyboards to acoustic guitars. When choosing a keyboard it’s important to get a quality keyboard that will last a while. Keyboards generally have a range of effects that can change the style of the sound it creates as well as even mimic other instruments like violins or drums. Grab a keyboard and take some piano lessons today!