Computer Design Copyright (c) 1988 Information Access Company; Copyright (c) Penn Well Pub. Co. 1988 March 1, 1988 SECTION: Vol. 27; No. 5; Pg. 41 LENGTH: 5072 words HEADLINE: Input technologies extend the scope of user involvement; Contains related articles on voice recognition and guidelines for input devices for 3-D display systems BYLINE: Williams, Tom BODY: Input technologies extend the scope of user involvement For applications ranging from drawing on personal computers to simulating complex real-world situations on supercomputers, innovative ideas for input devices focus on letting the user and the system work together in a natural, intuitive way. As computer systems and applications become increasingly complex, the need to simplify the human interface has also increased. Ironically, making computer systems simpler to use requires a great deal of technical effort. A PAGE 7 Computer Design (c) 1988 IAC sophisticated, real-time graphics display device, for example, is really just a way of simplifying data so humans can understand it. But the increasing volumes of complex data demand quantum leaps in processing power to bring it to the form users require--a picture on a screen. The richness and complexity of applications has led to the development of graphics-oriented user interfaces, such as windowing environments. Since graphics show objects conceptually related to the application, interaction with the system becomes faster and more intuitive than using a keyboard. Graphics-oriented user interfaces and applications--such as CAD and CAE--have spawned a variety of "point-and-pick" devices such as mice, joy sticks, track balls, knob boxes, digitizing tablets, light pens, thumb wheels and touch screens. In their simplest form, these devices let the user position a cursor or pointer, such as a stylus or a finger, in an X-Y coordinate space. At lease one button or switch is built in, letting the user select a function or an element from the display. Though the functionality of the X-Y point-and-pick devices may appear limited, each device has different characteristics that make one device more appropriate to a given application than another. The fact that they take on different physical forms indicates that different applications demand different approaches, even in simple X-Y input, and that the ideal input device hasn't yet been found. The two most popular input devices are the digitizing pad, or tablet, and the mouse. While both implement the simple point-and-pick function, the pad is an absolute positioner in that there's a 1:1 correspondence between a point on the surface of the pad and a point on the screen. A mouse is a relative positioner in that a point on the screen is selected by moving the mouse relative to its previous position. The mouse can be picked up and moved without affecting the placement of the cursor. As a result, only a small area of desk space is needed to use a mouse. Tablets can be fitted with a stylus, which is somewhat like a pen, or with a puck, which can be used much like a mouse, except for the fact that positioning is still absolute, rather than relative. The major advantage of tablets' absolute positioning characteristic is that it lets users enter data by tracing a drawing, for example. This feature has led to digitizers that are as big as a large drafting table for use in digitizing engineering drawings. GTCO (Columbia, MD) has even come up with a translucent digitizer that's 20 x 20 in. and can be used to digitize data from slides, which would normally be too small to digitize practically. And the Houston Instruments Division of Ametek (Austin, TX) is producing an accessory for its DMP-50 and -60 series of drafting plotters. The attachment rides on the pen carriage, scanning drawings to digitize and input them into a system. Another characteristic of digitizing tablets--and one in increasing demand by today's feature-rich applications--is the ability to define areas of the pad for pick functions. In addition, areas of the pad can be programmed as macros so users can get to a desired, commonly used command without traversing the menu. Also, other types of input devices will increasingly be called upon to help users move efficiently through large menu trees. As an example of what this can mean, the popular AutoCAD drafting package by Autodesk (Sausalito, CA) has a root menu with 16 choices, which leads to about 200 commands, many of which PAGE 8 Computer Design (c) 1988 IAC have their own option subset. Seiko Instruments (San Jose, CA) is reportedly developing a digitizing tablet with a built-in liquid crystal display (LCD) overlay. With a 640- x 400- pixel resolution, the LCD will approach moderate graphics screen resolution. One major advantage, according to senior product manager Michael Warner, will be the ability to dynamically download menus and macro names to areas of the pad for the user to select. The tablet will also let users customize menus and macros for their own convenience. Also, Warner adds, the user can add a grid to the pad to match the grid on the screen used in many drafting applications. Tree-structured menus appear to be the best way to organize complex applications. The way tree structures appear to the user and the way in which the user accesses the structure leave many options for the system designer to explore to construct a human interface. Traversing a menu tree, for instance, is a good way to learn how to use the application. And having the main command groups of the menu laid out on a tablet is an even better way. But when the user becomes familiar with the system, the menu tree gets in the way. At that point, users want to be able to go directly to the frequently used commands and parameters, and even customize the interface with macro commands tailored to their way of working with the system. Circumventing menu trees is becoming possible by even simple-appearing input devices, which are gaining intelligence in the form of microprocessors, programmability and two-way communication ability with the host system. In what appears to be yet another X-Y point-and-pick device, Felix from Lightgate (Emeryville, CA), illustrates this trend. Felix has a 6- x 6-in. base with a small handle that can be gripped by thumb and forefinger while the user's hand rests on the table. The handle, which has a button on it for the picking function, slides around in a 1 1/2-in. area on a proprietary grid of optical gates so users don't have to lift their hands to move the handle through the range of motion. Felix is an absolute positioning device in that each point of the handle's motion is mapped 1:1 to the screen coordinates. Putting absolute-positioning ability in a space that can be traversed without moving the hand has several advantages, according to Lightgate chief executive officer Victor Kley. "Displays are always absolute," he says. As a result, the absolute pointer lets the programmer work in straightforward coordinates without worrying about offsets or virtual coordinates, giving the user a more natural, intuitive feel when working with the screen. The intelligence of a device like Felix lets it use macros as well. The fact that the handle stops when it reaches the edge of the maneuvering area tells the user that he's at the edge of the screen and is no longer interacting with it. The corners and two ranges along each edge can be user-programmed as hot spots to contain frequently used menu commands or macros. Users can call these up on the screen, but they'll eventually be familiar enough with the hot spots that they could go to the appropriate edge area, invoke the desired macro and continue without removing their eyes from the screen. "The important thing about the hot spots," states Kley, "is that you can feel that corner. It's a kinesthetically stable place." Thus, invoking macros or other desired functions can be learned using lower brain-stem functions and can then become intuitive, much like riding a bicycle. As Kley says, "Your body motion can be quite complex while your main conscious function isn't involved." PAGE 9 Computer Design (c) 1988 IAC One type of input technology that's receiving a lot of attention is voice recognition. True speaker-independent recognition (SIR) has proven to be much more difficult than was originally supposed. Still, voice-recognition developments have led to system applications in which voice is the best form of input. These applications involve "busy hands, busy eyes," and include factory-floor, receiving-dock and airplane cockpit applications as well as command and control situations, according to James Ragano, executive vice-president and chief operating officer of Votan (Fremont, CA). Most of these applications require both alpha-numeric and voice command input. And most need a 100- to 200-word vocabulary. Although it's possible to nest groups of limited vocabulary sets to any depth by downloading, there's a limit to what the user can remember without being prompted from a display. Because of this, Ragano notes, "There's kind of a no-man's land between vocabularies that consist of a couple hundred words and 20,000 words." Two-thousand words is the threshold of what's considered necessary to create a voice-activated typewriter. Therefore, recognition systems of, say, 3,000 words don't seem practical. Most commercially available word-recognition systems are speaker-dependent recognition devices. SDRs must be trained by the user; users repeat the word to be recognized enough times for the system to build a template by analyzing the frequency and timing of the spoken word. Recognized words are then sent to a lookup table to output appropriate codes to the computer system. According to Ragano, the device with the greatest potential isn't voice-activated typewriter or a free-form dictation machine, but rather an SIR system that can work with a fairly limited vocabulary over the telephone and recognize words out of continuous speech rather than separated words. The telephone limits practical voice recognition, however, because of its limited bandwidth. Also, because the quality of the voice lines is unpredictable, the system needs a high degree of noise immunity. Ragano believes the advantages for telemarketing systems, on-line help and emergency services would be enormous if such a technology became economically feasible. A system that could reliably work over noisy lines and recognize regional accents and even slurred speech represents a formidable set of technology hurdles. But Ragano hints that solutions may be near at hand. One fundamental problem, he notes, is that we've tried to recognize speech by constructing an analytical model rather than modeling the way the human ear and the nervous system process data. Also, it was once assumed that interfacing speech recognition with the system was a matter of matching recognized words to a lookup table to invoke the proper commands and data input. But we now know that speech is a much more complex human interface issue. Applications must be "voicified" or structurally modified to work in a way that's natural to the user. In applications where it's desirable to prevent users from having to learn commands or from getting their hands on possible breakable input devices, such as in public telephone booths, touch-screen technology can be one way to provide direct interaction with a display. Touch-screen systems from companies such as Carroll Touch (Round Rock, TX), Elographics (Oak Ridge, TN) and Tektronix (Beaverton, OR) provide several technology choices, including infrared arrays that criss-cross the screen and detect the location of a finger or stylus, capacitive and resistive overlays, and surface-acoustic wave technologies. PAGE 10 Computer Design (c) 1988 IAC Resolutions and hardware characteristics vary, but touch technology is a simple way to lead inexperienced users through a system that uses graphics-based menu trees. Another application area in which touch technololgy is used is complex-control situations in which the display not only can provide touch areas to invoke system functions, but also can integrate them with graphics that schematically display the system (such as a refinery or factory) under control of the computer. Also, the increased resolution of touch systems makes them suitable for picking functions as well as for adjustable controls such as virtual sliding potentiometers displayed on the screen and controlled by rubbing the finger in the desired direction of adjustment. Attempts are being made to adapt input devices originally designed for the two-dimensional world to work with three-dimensional displays. This would make it possible to rotate the shaft of a joy stick to move the cursor in the Z axis and take advantage of the ability of touch screens using surface waves to sense the pressure of a finger. This pressure-sensing ability can be used to move in the Z axis as well. GTCO has developed a stylus for use with its digitizing tablets that can sense pressure to five bits of resolution (32 different levels). While this could be used to a limited degree for Z-axis control, it's intended to give graphic artists more control of line thickness or color. Lightgate is developing a version of the Felix that will include a wheel on the side of the unit that will control the Z axis . And several companies, including Hewlett-Packard (Palo Alto, CA), Seiko and GTCO, provide knob boxes with nine controls. Six of the knobs are used for manipulating ofjects in all three axes and degrees of rotation around each axis and the three remaining knobs are used for pan, scroll and zoom operations. There are a number of deficiencies with today's input devices, such as the inability to switch a device between absolute and relative positioning as well as the inability to move intuitively in 3-D, according to Michael Sleator, a member of the technical staff at Ardent Computer (Sunnyvale, CA). He uses both a mouse and digitizing tablet while developing single-user supercomputers that will be used extensively for 3-D graphics and interactive real-time simulation in fields such as computational fluid dynamics and molecular modeling. Sleator shifts between a CAE system with a built-in digitizer tablet and another with an optical mouse. The main problem with shifting between the two is that because the mouse uses relative positioning and the tablet uses absolute positioning, it's difficult for the user to remember where each one should be, relative to the other. There should be some way to switch the tablet between absolute and relative mode, according to Sleator. "One way to do this would be to have a switch to tell whether [the puck or stylus] is down, then software to switch between modes," Sleator says. Also, he believes there should be a way to turn off one of the axes, letting the user move the cursor along the Y axis while the X axis position remained the same--even if the user's hand wavered. For 3-D graphics, there aren't many appropriate input devices, notes Sleator. "People accept tablets and knob boxes--both of which are incredibly primitive." He adds that the need for better interactive devices for use with 3-D graphics is apparent in molecular modeling problems--where a basic problem is trying to "dock" two complex shapes together. If a screwdriver, a screw and a piece of wood are items in a display, for example , it should be possible to use some PAGE 11 Computer Design (c) 1988 IAC type of input device to pick up the screw, place it in the wood, pick up the screwdriver and then turn the screw, according to Sleator. How such an action would be performed is being explored. One of the first requirements for a 3-D input device that feels natural to the user is that the device must be able to track the position of the user's hand as it moves in 3-D space, and then return position coordinates accurately to the system. In fact, tracking 3-D objects requires not only knowledge of the object's X, Y and Z coordinates, but also knowledge of the orientation of--and rotation about--the object on each axis. These factors are referred to as the "six degrees of freedom," and include X, Y, Z, roll, azimuth and elevation. To manipulate the 3-D representation of an object, the user moves his hand or a pointing device (such as a 3-D mouse or stylus) around an invisible model in space, as the cursor moves around a visible 3-D image on the screen. This raises the question of ergonomics--how natural is it to manipulate 3-D images in this way? Moving one's hands about in empty space without support could be quite fatiguing, yet resting the hand on one spot and manipulating something like a joy stick with three dimensions (X, Y and a twist for Z) would be less intuitive than using the hand in full motion. This dilemma is being explored. One recent development that addresses this question is the Spaceball from Spatial Systems (Milsons Point, NSW, Australia). In the Spaceball, a knob the size of a tennis ball sits on a stem. It reacts very little when pushed or twisted, but senses pressure and torque. Thus, the Spaceball is an isometric device in that the user doesn't move his hand to control the cursor, but simly pushes and twists on the stationary knob. There's a pick button on the knob where the user's finger can rest, and there are eight function buttons on the knob. The user's arm rests on a chaise-lounge-shaped support to minimize fatique. "The knob feels like a reasonably solid ball with a slightly spongy feel," comments John Hilton, Spatial Systems' technical director. The Spaceball implements the six degrees of freedom needed to completely position a 3-D object. It's programmable so that each axis or rotation on an axis can be selectively disabled. Low-level interfacing to existing applications is straightforward, according to marketing director Ian Roberts. "The easiest way to implement it is to say that it replaces a dial box," he says. In other words, the basic software driver closely resembles a driver used with a nine-knob dial box. Roberts notes, however, that the Spaceball goes beyond traditional pointing devices in that it's programmable and communicates with the host. On one level, the user can assign functions to the buttons on the device to perform such functions as selectively turning off an axis or executing macros. On another level, the programmer can build functions into the application that selectively download routines to the buttons or let the user customize the device. Spatial Systems is currently developing a library of C-language subroutines that will let the application programmer send commands and parameters to the ball. The level of interfacing to the application will thus be quite flexible. A user can interface with existing applications at the driver level and take advantage of the Spaceball's native modes, such as emulating a 2-D mouse, or can write macros to suit his own needs. A software OEM can even build routines into the application that specifically use the Spaceball's programmability. Mouse emulation is useful when the user wants to shift into a 2-D mode to move PAGE 12 Computer Design (c) 1988 IAC around a menu and pick functions. Turning off axes often makes it easier to position an object on a specific axis. For instance, if an object is already positioned where the users wants it on the X or Y axis, the user could turn off those axes and their associated rotations, and position the object precisely on the Z axis. The basic requirement for a device that can return point coordinates in 3-D space as well as axis orientation back to the computer has been addressed by Polhemus Navigation Sciences (Colchester, VT). The 3Space Tracker unit consists of a fixed transmitter that emits electromagnetic pulses that provide a reference for the position of a sensor. The sensor is attached to a hand, a head or an object to be tracked in 3-D space. The transmitter, which is typically located in some fixed reference position, contains three coils positioned at right angles to each other. The coils act as antennas and emit three pulses in succession--one for each axis. The sensor is a receiving unit that also has three antenna coils oriented at right angles to one another. Three pulses from the source unit induce nine pulses of current in the sensor, since each pulse from the source is received by all three coils in the sensor. The 3Space Tracker's control unit calculates position and orientation from the nine current values. Sets of three pulses are emitted at a rate of 60 times/s for a spatial resolution of 0.03 in. and an angular resolution of 0.1 degrees . The 3Space Tracker is finding its way into numerous applications, such as biomedical and neurological research. In one application, a sensor attached to a patient's hand is used to track his movements in response to stimuli for assessing conditions such as damage from a stroke and the recovery progress. Polhemus has also developed a 3-D digitizer using the 3Space Tracker. With the source unit mounted under a table and a sensor in the form of a stylus, the system can be used to input 3-D data points from a physical object on the table. A more creative use of the 3Space tracking technology lies in the fact that it lets the user work in a virtual environment, according to Tom Knoflick, Polhemus marketing manager. Virtual environments are simulations developed on supercomputers that create an artificial reality so that users feel as if they're moving through a building or a molecule, or as if they're robots working outside a space station. There are even attempts to graphically model a way to move through such abstractions as mathematical equations. In producing virtual environments, the 3-D input device is used to form a close link between the user and the display. One step toward such virtual environments is the Dataglove by Visual Programming Language Research (Redwood City, CA). The Dataglove is a glove worn by the user to create a virtual hand on the display screen with which the user can manipulate items in the display. A sensor from the Polhemus 3Space Tracker mounted on the back of the hand provides the system with position and orientation information on the user's hand. The glove contains another entire subsystem to model the position of the fingers and thumb. For each joint in the hand, there's a fiberoptic loop that runs from the control board near the wrist to the respective joint and back to the control board. On one end of each loop is a light-emitting diode, and at the other is a phototransistor. The jacket of the fiberoptic cable is treated at each joint where the finger flexes, letting light escape proportionally to the degree of PAGE 13 Computer Design (c) 1988 IAC flexure. Thus, the amount of light sensed by the phototransistor is inversely proportional to the degree of flexure of that joint. The combination of the position information from the Polhemus Sensor and the flexion sensors in the glove is integrated to produce a virtual hand on the display screen in the same screen position and with the same finger positions as those of the user. Directly manipulating objects in the display, such as grabbing a piece of a molecule and moving it, is only one level at which the glove can be used, notes Jean-Jacques Grimaud, president of Visual Programming Language (VPL) Research. A much more interesting level, he contends, is the use of the virtual hand to manipulate virtual tools designed to be approriate to specific applications. This idea is analogous to the screwdriver example from Ardent's Sleator. In 2-D positioning, people use their hands to move a mouse to an X-Y position, and then use a button to select a "tool," whether it's a menu item or an option box in a windows system, to manipulate an object set that represents the application, Grimaud explains. "The idea of the Dataglove," he says, "is to design tools in software that are appropriate to the application and can be manipulated by hand." Possible applications include rehearsing a surgical operation by manipulating a virtual scalpel within a volumetric scan of a part of a patient's body. Another use is telerobotics--the command and control of remote physical devices by simulating the robot's view. Virtual robots would let the user work in environments ranging from giant automated construction equipment to microsurgery. Users would operate in a virtual environment scaled to their real world, and the system would translate the user's actions to the scale of the application's real world via the computer's virtual world. Researchers at the University of North Carolina (Chapel Hill, NC) and at NASA's Ames Research Center (Mountain View, CA) are working with a head-mounted display that changes as the user moves his head. The display has half-mirrored surfaces so the projected display appears to the wearer when the ambient light is sufficiently low. A Polhemus transmitter is attached to the back of the head-piece to send position and orientation to the computer, which pans and scrolls the head-mounted display as the user moves his head. One problem in truly representing reality is that humans can look around by moving their eyes as well as their head. Attempts have been made to use laser devices to track the movement of the eyes as additional feedback to the system. Laser eye-trackers have been used to some extent in psychology research, but according to Scott Fisher, research scientist at NASA Ames, people don't like the idea of having a laser, even a low-power infrared laser, constantly shining in their eyes. So other than a few arcane military applications, the future of laser eye-trackers appears to be limited. The University of North Carolina uses the head-mounted display with a treadmill in a system that simulates walking through a building. The blue-prints of a campus building were digitized, letting a user literally walk through the building in a "virtual reality," says John Hughes, research associate at the university's computer science department. NASA Ames is developing telerobotics applications for a system that includes the head-mounted display. Such applications are aimed at controlling a remote robot through the eyes of the robot itself, which is planned to be used outside of the planned U.S. space station. In addition to including the head-mounted display, the system also uses the Polhemus 3Space Tracker system and two PAGE 14 Computer Design (c) 1988 IAC Datagloves. In addition, there's a voice input device. As users move their hands, they can see virtual hands moving in front of them in the "data space" created by the system. The University of North Carolina is also experimenting with "force feedback," which is in demand by people doing molecular modeling. When a user is positioning a complex molecule to see if it can bond to another molecule, for example, the criteria isn't only the shape of the place he's trying to put it, but also the chemical properties of attraction or repulsion. Molecular modelers would like to see these forces at work. When they try to push a molecule into a place where it doesn't belong, they want to feel it pushing back at them. The university has acquired a robotic arm that's designed for handling hazardous materials, and it's working on adapting the arm as a six-position joy stick with force feedback, according to Hughes. The motors on the arm have been modified so that rather than running duplicate arms on the other side of a glass barrier, the motors feed back resistance to the operator in response to data from the computer. The motors--which can lift 17 lb--also had to be weakened so the arm could't reach out and hit the user. Hughes stresses that the experimental arm is no longer a robotic arm, but a data input and feedback device that's trying to tell the user something about the data in the computer model. If using the arm works, more user-appropriate physical input devices can be designed. The joy stick, mouse and tablet were the first successful alternatives to keyboard user input. As applications continue to grow richer in features and in their resemblance to real-world environments, the industry can expect increased activity in finding natural ways for humans to interact with machines. In fact, VPL Research intends to extend the concept of the Dataglove to a Datasuit to be unveiled later this year. How much more involvement in the computer environment can we expect? PHOTO : For digitizing transparent films, the Digipad 2020r from GTCO lets slides, which would PHOTO : normally be too small to be digitized practically, be projected on the back of a PHOTO: transparent 20-x20-in. digitizing tablet. Other types of transparent digitizers can be PHOTO : used with standard light tables for large-format films. PHOTO : Unlike a mouse, which is a relative positioner, the Felix input device from Lightgate is PHOTO : an absolute positioner and features a 1:1 correspondence between points in the PHOTO : 1 square inch motion area and the entire display area. The handle slides around freely so PHOTO : users don't have to lift their hand to move the handle through the full range of its PAGE 15 Computer Design (c) 1988 IAC PHOTO : travel. Dots on the edges and corners indicate stable areas that may be programmed with PHOTO : macro commands or other functions. PHOTO : The Spaceball from Spatial Systems lets users completely position a 3-D object. It's PHOTO : programmable so that each axis or rotation around an axis can be selectively disabled. The PHOTO : ball has little give when pushed or twisted, but senses pressure and torque. Thus, PHOTO : Spaceball is an isometric device in that users don't move their hands to control the PHOTO : cursor, but simply push and twist on the stationary ball. Eight programmable function PHOTO : buttons can be reached without removing the hand from the control ball, which also PHOTO : contains a pick button. PHOTO : The Dataglove from VPL Research lets users move and flex their hand in free space. The PHOTO : glove is represented by a virtual hand on the screen in the same position and state of PHOTO : flexure as the user's hand. The user is thus able to manipulate virtual tools appropriate PHOTO : to the application, much as if he were using such tools in the real world. PHOTO : A system for moving within artificial realities developed at NASA's Ames Research Center PHOTO : incorporates a head-mounted display, which changes as the user moves his head. The PHOTO : display has half-mirrored surfaces so that the projected display appears to the wearer PHOTO : when the ambient light is sufficiently low. With the Datagloves from VPL Research, the PHOTO : user can see virtual hands moving in front of him in the "data space" created by the PAGE 16 Computer Design (c) 1988 IAC PHOTO : system. The system also includes one Polhemus transmitter each for head and hand position PHOTO : and orientation, and a voice-recognition subsystem. GRAPHIC: Photograph PRODUCT-NAME: Felix (I-O device), usage; Spaceball (I-O device), usage; Dataglove (I-O device), usage COMPANY: Lightgate, manufactures; Spatial Systems, manufactures; Visual Programming Language Research, manufactures; SIC: 3577