| Publications Page | HITL Home |

Hoffman, H.G. (1998). Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. Proceedings of the IEEE Virtual Reality Annual International Symposium '98, Atlanta GA, p. 59-63. IEEE Computer Society, Los Alamitos, California.

Physically Touching Virtual Objects Using Tactile Augmentation Enhances the Realism of Virtual Environments.

This study explored the impact of physically touching a virtual object on how realistic the VE seems to the user. Subjects in a "no touch" group picked up a 3-D virtual image of a kitchen plate in a VE, using a traditional 3-D wand. "See and touch" subjects physically picked up a virtual plate possessing solidity and weight, using a mixed-reality force feedback technique. Afterwards, subjects made predictions about the properties of other virtual objects they saw but did not interact with in the VE. "See and touch" subjects predicted these objects would be more solid, heavier, and more likely to obey gravity than the "no touch" group. Results provide converging evidence for the value of adding physical qualities to virtual objects. This study is the first to empirically demonstrate the effectiveness of mixed reality as a simple, safe, inexpensive technique for adding physical texture and force feedback cues to virtual objects with large freedom of motion. Examples of practical applications are discussed.

Keywords: virtual reality, tactile feedback, force feedback, calibration, realism

Introduction.

Most commercially available VR systems do not include tactile or force feedback. When the typical VR user reaches out to pick up a virtual object, their cyberhand goes into/through the object. Such virtual objects have no solidity, no mass, and often don't obey the rules of gravity (i.e., they float in the air when dropped), detracting from the realism of the VE. A number of research centers have developed innovative computer-simulated force feedback techniques, but despite promising progress, tactile feedback is lagging behind visual and auditory VR input technologies [1]. Tactile augmentation [2], touching real objects while in virtual reality, is an effective alternative mixed reality [3] technique for introducing tactile cues.

The present study employs tactile augmentation to explore the impact of physically touching one virtual object on user's predictions about the properties of other virtual objects and the "laws of nature" obeyed in a virtual kitchen. Subjects were randomly assigned to one of two conditions. Those in the "no touch" condition picked up a 3-D virtual image of a kitchen plate, using a traditional 3-D wand to control their cyberhand. "See and touch" subjects physically picked up the virtual plate, see Figures 1 and 2. Their real hand grabbed a real ceramic plate in the appropriate spatial location. The VR system tracked the position of the real plate (using a position sensor) such that any change in position or orientation of the real plate was mimicked by the virtual plate seen in VR. As a result of the brain's propensity to unify disparities in the two modalities of input and for vision to dominate [4], the visual virtual object captured the tactile properties of the real object, creating the illusion of a virtual object with the properties of the real object , e.g., "cyberheft". Subjects later made predictions about the properties of other objects they saw but did not interact with in the virtual world. I predicted that subjects in the "see and touch" condition would rate the teapot, walls, and countertop as more solid, and rate the teapot heavier, and more likely to obey the laws of gravity than subjects in the "no touch" condition. Examples of practical applications of tactile augmentation are discussed.

Experiment 1

Method

Subjects.

Nineteen students from the U. of Washington participated in the 20 minute experiment.

Materials and equipment.

A real kitchen plate, 11" in diameter, was modeled in 3-D and texture mapped with a digitized texture from the real plate and placed on a small white table. The virtual image was scaled using a mixed reality ruler (an objective calibration technique developed for this study, see appendix) such that pilot subjects indicated a close correspondence between what was seen in VR and what was felt when they touched the real plate.

The VR system consisted of a Division ProVision 100, coupled with a Division dVisorTM HMD with the following FOV: 40 degrees vertical, 105 degrees horizontal combined across two eyes, and 40 degree horizontal overlap. A polhemous sensor attached to a fingerless bicycle glove (right hand) was used to control cyberhand positions. A second sensor attached to the bottom of the real plate controlled movements of the virtual plate (See Figures 1 and 2).

Design and Procedure.

A between-subjects experimental design was used. Each subject was randomly assigned to either a "see only" or a "see and touch" condition.

Each subject donned an HMD and viewed Division LTD's KitchenWorld demo. When subjects in the "see and touch" condition placed the plate on the real table top, they saw the virtual plate rest on the countertop in VR.

Subjects were told that they would see a virtual plate, which they were to pick up with their cyberhand. Subjects in the "no touch" condition were instructed on how to pick up objects by immersing their cyberhand into the object, and pulling the trigger button of the 3-D mouse to pick the object up. Subjects in the "see and touch" condition were instructed to reach out with their cyberhand and pick up the virtual plate by grabbing the real plate with their real hand.

After the VR phase, subjects filled out a brief questionnaire. They were instructed to "Please make predictions below regarding the properties possessed by the virtual kitchen you experienced" (given a scale marked 1 2 3 4 5 6 7).

A. In the virtual world, you saw a tea pot on the countertop. How solid was the teapot? 1 = only visual, not solid, 7 = as solid as a real teapot

B. In the virtual world, how solid were the walls of the kitchen? 1 = only visual, not solid, 7 = as solid as a real wall

C. In the virtual world, how solid was the countertop on which the plate resided? 1 = only visual, not solid, 7 = as solid as a real countertop

D. In the virtual world, if you picked up the teapot, how much would it weigh?

1 = only visual, no weight, 7 = as heavy as a real teapot

E. To what extent do you predict that the teapot would obey the laws of gravity?

1 = not at all, 7 = same gravity as in the real world

Results

One mean was calculated for each subject and used in the analysis. Subjects in the "see and touch" group gave higher ratings than subjects in the "no touch" group (mean ratings = 5.0 vs. 3.2 respectively). A Wilcoxon, signed-rank test (a non-parametric t-test) showed a highly significant difference between the two groups, Z = 2.70, two-tailed p = .006. This comparison (labeled "means"), and an item analysis are shown in Figure 3. The pattern of higher ratings for the "see and touch" group compared to "see only" group was the same for each of the five questions (A,B,C,D and E).

Discussion.

Results provide converging evidence for a growing literature showing the value of adding physical qualities to virtual objects (e.g., [2,5]). This study also demonstrates the effectiveness of tactile augmentation as a technique for adding texture and force feedback cues to virtual objects.

When subjects enter KitchenWorld, they have to adapt to this new environment. Adaptation likely involves assessing what rules from the real world apply in this strange virtual world. The present study shows that the experience they have with the first virtual object they interact with can have a large influence on their perception of the properties of other virtual objects, and the "laws of nature" obeyed in that VE. The "cyberheft" of the plate experienced by subjects in the "see and touch" group led them to expect other virtual objects to have more realistic properties. Ideally, each virtual object that subjects are likely (or allowed) to touch or pick up will also be a mixed reality object. Achieving this ideal would be relatively easy with wireless position sensors and systems allowing large numbers of position sensors (e.g., Ascension flock of birds position tracking system).

Future research: Practical applications of tactile augmentation.

Three practical applications that might benefit from the use of 'tactile augmentation" are described below.

Use of tactile augmentation to maximize burn pain reduction.

Hospitalized burn patients typically experience severe to excruciating pain during wound care (cleaning etc.), despite treatment with potent morphine-based analgesics. VR (see Figure 4) appears to help by distracting patients (especially children) from burn pain [6,7]. We speculate that adding touch cues to virtual objects could make the objects more attention grabbing, increasing the effectiveness of the VR distraction treatment.

Use of VR for treatment of spider phobia.

Encouraged by the successful application of virtual reality to the desensitization of fear of heights [8], my colleagues and I [9] recently explored the use of virtual reality for treating spider phobia. We took advantage of a mixed reality spider in the treatment of a severe spider phobic. The subject interacted with a brown virtual spider (see Figure 5). The real-world counterpart of the brown spider, used for tactile augmentation, consisted of a furry palm-sized replica of a Guyana bird-eating tarantula. As the patient reached out with her cyberhand to explore the virtual spider, her real hand explored the toy spider attached to a polhemus position sensor. The virtual spider now felt furry, and had weight ("cyberheft") and any movement of the toy spider caused a similar movement of the virtual spider.

Being able to touch the virtual spider dramatically heightened the intensity of the fear/anxiety experienced by our patient, a manipulation important for successful treatment using systematic desensitization. Desensitization to the virtual spider generalized to real spiders. Our patient made fast long term progress. One year after treatment, she is no longer phobic of real spiders. Two additional clinical-level spider phobics have now been successfully treated using VR exposure therapy with tactile augmentation. We speculate that tactile augmentation helped generalization of training from virtual spiders to real spiders by blurring the distinction between real and virtual.

Virtual-reality Monitoring

The essence of immersive virtual reality is the sensation users have that they are "there" in another place. Users have a sense that they "go into" the 3-D, immersive, computer-generated environment. They become involved in events in VR, and these experiences leave memories. Memories for events that occurred in VR constitute a new source of memories, different from those traditionally studied (real events, imagined events, or dreams). Virtual-reality monitoring [10], a variation of reality monitoring [11] is the decision process by which people discriminate and sometimes confuse memories of real and virtual events.

In a recent study, Hoffman [12] exposed subjects to 24 common objects (e.g., apple) one at a time. Some objects were seen in VR, others were seen in the real world. A week later, subjects returned to take a source memory identification test (36 items on test). They were shown the name of an object (e.g., spatula), and had to decide whether they saw it in the real world during the study phase, in the virtual world, or if it was new.

In second experiment, subjects are able to physically touch the virtual objects using tactile augmentation. A real object (e.g., a rubber ball) is placed within the participant's grasp at the location of the virtual object to explore the impact of adding tactile cues on source memory identification accuracy. I predict that being able to physically touch virtual objects will result in the formation of "chimeric" memories that are part real (the touch part) and part virtual (the visual part). The results reported in the present paper show that being able to physically touch virtual objects can make the virtual objects and the VE much more realistic. This finding leads to a prediction for performance in a virtual-reality monitoring task. Because mixed reality memories will be more similar to real memories than untouched visual-only virtual objects, people will be more likely to confuse real and virtual objects in the "see and touch" condition than in the "vision only" condition.

Memory source confusions may serve as a human factors measure of how realistic users find the virtual experience. For VEs modelling the real world, the greater the fidelity of the virtual environment to the real world, the more likely subjects are to confuse the two, the higher the quality of the VR system. Phenomenological qualities associated with virtual memories, which "tip off" the user that these happened in VR, are targets for improvements in VR systems. Like other source monitoring tasks [see 13, 14], virtual-reality monitoring may prove valuable for understanding human memory (e.g., age-related declines in memory performance), and to "cybercognition", the study of how humans think in immersive and non-immersive computer-simulated environments.

Conclusion.

Tactile augmentation differs from VR systems that involve expensive, computer generated force-feedback [15, 16]. Admittedly, there are numerous applications where computer-generated force feedback devices are required (e.g., telerobotics where sensory information from the remote location needs to be made available to the user). When appropriate, a tactile augmentation system cost very little money, and the physical textures of the real objects (e.g., the fuzzy feel of a peach) are hard to reproduce in computer simulations. Furthermore, input from real objects is computationally inexpensive, safe, and allows large freedom of motion. And studies using this shortcut may inform the design of computer-generated force feedback devices.

Acknowledgments

Research support by AFOSR grant #F49620-93-1-0339, as well as Tom Furness and the HITLab industrial consortium. Thanks to Joris Groen, Ari Hollander, Bill Winn, and Max Wells and Scott Rousseau and reviewers. Copywrite Permissions: Fig 1 & 2 from H. Hoffman, Fig 4 from G. Carrougher, U.W., Fig 5 from Mary Levin, U.W.

References

1. Burdea, G.C. (1996). Force and touch feedback for virtual reality . New York : John Wiley & Sons.

2. Hoffman, H.G., Groen, J., Rousseau, S., Hollander, A., Winn, W., Wells, M., and Furness III, T. (1996). Tactile augmentation: Enhancing presence in inclusive VR with tactile feedback from real objects. Presented at the meeting of the American Psychological Sciences, San Francisco.

3. Milgram P., and Kishino, F. (Dec., 1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, Vol E77-D.

4. Welch, R.B., & Warren, D.H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638-667.

5. Burdea, G., Richard, P., and Coiffet, P. (1996). Multimodal virtual reality: input-output devices, system integration, and human factors. International Journal of Human-Computer Interaction. 8, 5-24.

6. Hoffman, H.G., Doctor, J.N, Patterson, D.R., Weghorst, S., & Furness, T. (submitted). Use of virtual reality for adjunctive treatment of pediatric and adolescent burn pain: A case report.

7. Hoffman, H.G., Doctor, J.N., Patterson, D.R., Carrougher, G.C., Taylor, W., Weghorst, S., and Furness, T. III. (1998). VR for burn pain control during wound care. Paper to be presented at Medicine Meets VR 6, San Diego CA.

8. Rothbaum, B.O., Hodges, L.F., Kooper, R., Opdyke, D., Williford, J. & North, M.M. (1995). Effectiveness of virtual reality graded exposure in the treatment of acrophobia. American Journal of Psychiatry, 152, 626-628.

9. Carlin, A.S., Hoffman, H.G. & Weghorst, S. (1997). Virtual reality and tactile augmentation in the treatment of spider phobia: A case study. Behaviour Research and Therapy, 35, 153-158.

10. Hoffman, H.G., Hullfish, K.C., and Houston, S. (1995). Virtual-reality monitoring. Proceedings of the Virtual Reality Association International Symposium, 95. (pp. 48-54). IEEE Computer Society Press, Los Alamito, CA.

11. Hoffman, H.G. (in progress). Virtual-reality monitoring: Separating real from virtual memories.

12. Johnson, M.K., & Raye, C.L. (1981). Reality Monitoring. Psychological Review, 88, 67-85.

13. Dywan, J. & Jacoby, L. (1990). Effects of aging on source monitoring: Differences in susceptibility to false fame. Psychology and Aging, 5, 379-387.

14. Brooks, F.P. Jr., Ming, O-Y, Batter, J.J., Kilpatrick, P.J. (1990). Project GROPE: Haptic displays for scientific visualization. Computer graphics, 24, (4), 177-185.

15. Tachi, S., and Yasuda, K. (1994). Evalutation experiments of a teleexistence manipulation system. Presence: Teleoperators and Virtual Environments, 3, 35-44.

16. Robinett, W. & Rolland, J.P. (1992). A Computational model for the stereographic optics of a head-mounted display. Presence: Teleoperators and Virtual Environments, 1, 45-62.

17. Roscoe, S.N. (1984). Judgments of size and distance with imaging displays. Human Factors, 21(6), 617-629.

18. Rolland, J.P. Gibson, W. & Ariety, D. (1995). Towards quantifying depth and size perception in virtual Environments. Presence: Teleoperators and Virtual environments, 4, 24-49.

Appendix

In order for this technique to be effective, the size and shape of the virtual images must correspond reasonably closely with the size and shape of the real object subjects touch (orthostereoscopy, see [16]). The brain tends to overlook small discrepancies in an attempt to make sense out of the world, but if the mismatch between real and virtual size/shape becomes too large, the illusion of unification breaks down: participants realize that what they are touching and what they are seeing are two different objects [see 4]. Unfortunately, distortions in the optics of the HMD make objects in VR appear "smaller than life" [17,18]. To calibrate the size of the virtual objects to the actual real size, I put a virtual ruler into the VE. Using tactile augmentation, I then scaled the virtual ruler so that it was the same length as a real ruler. A position sensor attached to my index finger controlled the position in VR of a virtual pointer. Placing my position tracked real finger at zero or 12" on the real ruler put the virtual pointer at zero or 12" respectively on the virtual ruler. I placed the virtual plate on the mixed-reality ruler, with the left edge of the virtual plate at zero and measured the virtual plate diameter. I now measured the actual diameter of the real plate (no VR needed to do this). This provided the ratio needed to re-scale the virtual object so that it matched the real object in size almost exactly.