A Virtual Retinal Display For Augmenting Ambient Visual Environments

by Michael Tidwell

[Previous Chapter][Table of Contents]

Chapter 9:Future Work and Conclusions

9.1 Conclusions

A full color (RGB), 40 [deg.] horizontal field, 30[deg.] vertical field, 640 horizontal by 480 vertical picture element (VGA) retinal scanning display has been demonstrated and characterized for this thesis. Experiments show that individual picture elements and scan lines are resolvable in the display which indicates that the resolution is truly 640 by 480 elements. The display also performed effectively in a see-through mode exhibiting sufficient brightness to be viewed against a normal room illumination background. The technology in this thesis should be extendible and adaptable to an SVGA (1280 by 1024) display.

9.2 Areas for Improvement

Performance limiting technical issues were presented in the previous chapter. Remaining technical problems and potential solutions will be discussed in this chapter including work in the areas of light sources (Section 9.3) , scanning technology (Section 9.4), viewing optics (Section 9.6), and visual cues (Section 9.7). Ways to improve the small exit pupil of the system are also discussed in Section 9.5.

It is necessary for any future work to include the needs of the end user. The applications are diverse and some are highly technical. For example, if a major application for an augmented vision system is in anesthesiology and, by chance, anesthesiologists perform better using yellow augmented displays instead of RGB augmented displays, then the sooner it is known the better. Since little work has been done in the area of see-through head mounted displays for applications other than the military, more work is needed to find out what a user needs from this type of display and why.

9.3 Light Sources - Light Emitting Diodes

An important characteristic of any light source for the VRD is modulation bandwidth or more precisely, the modulation transfer function. It is desired that the light source maintain a high modulation depth at high frequencies. There is indeed a tradeoff between modulation bandwidth and luminance in most LED structures. Although lasers with external modulators are presently available, directly modulated semi-conductor lasers are not presently available in blue and green. Although semi-conductor laser are a potential future solution to the problem of unsatisfactory light sources, light emitting diodes may offer a more viable solution in terms of cost and size.

The light sources for this project are a 650 [nm] semiconductor laser, a 543.5[nm] HeNe, and an air cooled 488[nm] argon laser. These light sources can be considered neither inexpensive nor small (or lightweight). Light emitting diodes (LED's) may offer an alternative to bulky and expensive lasers. LED's of sufficient intensity could be coupled to an optical fiber. A different coupling scheme from the coupling scheme described earlier, however, would be required. If the emission area an LED were small enough (on the order of 5[mm]) and of the correct geometry (namely circular), an LED could be shined directly onto the horizontal scanner with a minimum of corrective optics. It should be noted that an LED with a traditional rectangular emission area would have an astigmatic wavefront and require corrective optics to compensate for the astigmatism. A rectangular emission aperture causes orthogonal planes of the wavefront to have different focal points (i.e. astigmatism).

9.4 Scanning Technology

9.4.1 Overview of Scanner Development

The optical beam scanner for this project is a novel is a novel moving mirror device based on a resonant electromechanical system [45]. The important scanner characteristics relevant to the VRD are scan angle (or scan deviation), scanner aperture (beam size at scanner), scanner speed, and also size and weight. The Mechanical Resonant Scanner (MRS) has a small form factor and is light weight at 2 [cm3] and approximately 2 [oz.]. The scanner design, as partly shown in Section 6.2.2, is a complex problem. More work is needed to reduce scanner volume while increasing scan frequency and mechanical deviation.

9.4.2 Mechanical Resonant Scanner Phase Detection

The frequency of the scanner is temperature dependent and may drift. One way to curtail the effects of drift on display quality is to adjust the phase difference between the light modulation and the scanner position by adding delay to the light modulation. If the phase difference is to be adjusted automatically, the phase difference between the light modulation and the scanner position over time (or scanner timing) must be known. Although the modulation timing is easily determined in the electronics, the scanner position over time is more difficult to predict. One idea for measuring scanner phase involves detecting back electro-motive forces in the scanner coils [46]. Once the scanner timing can be detected, an electronic circuit will continually reduce the phase difference between the light modulation and scanner timing by adjusting the light modulation delay.

9.4.3 Sinusoidal Scan Correction

Since the horizontal scanner is a mechanically resonant device, the deflection of the scanner mirror is sinusoidal over time. Therefore, the scanner is moving with greater speed at the center of its motion (through the at rest position) and slower at the extremes of its motion. Consequently, a constant modulation rate results in a distorted image which is expanded in the middle of the horizontal scan and compressed in the middle. A solution for this problem is too modulate the pixels at a faster rate in the middle of the display. Much of the noticeable distortion occurs at the outermost 5% - 10% of the display. Blanking the outermost 5% - 10% may make sinusoidal correction easier.

9.4.4 Increasing the MRS Scan Frequency to Super VGA

The SVGA graphics format (1280 1024 picture elements) is the next step in terms of improving resolution for the Virtual Retinal Display technology. A bi-directional horizontal scanner must have a scanning frequency of 30.72 [kHz] to achieve SVGA resolution with a 60 [Hz] frame rate. Initial experiments indicate that a 30.72 [kHz] scanning frequency is obtainable from the MRS with only slight reduction in optical deflection [46].

9.4.5 Improving Mirror Surface Quality

By experimenting with different scanner mirrors, the scanner mirror was found to be the main cause of the astigmatism afflicting the optical performance of the system. However, improved polishing techniques have proved to significantly decrease the astigmatism introduced to the scanned wavefronts by the scanner mirror [46]. A Virtual Retinal Display using an improved scanner mirror has recently been built which provides improved performance to the extent that individual pixels are easily resolved.

9.5 Exit Pupil Size

9.5.1 Scanner Aperture

It is evident that a larger exit pupil is required for this display to be practical in most applications. One way to achieve a larger exit pupil with the MRS approach is to increase the mirror size (see Section 6.2.2). However, it is a difficult mechanical engineering problem to increase scanner size and speed simultaneously. In the mechanical system of the MRS, increasing the mirror size tends to decrease scanner frequency and/or deflection.

9.5.2 Exit Pupil Duplication

One way to increase the apparent exit pupil may be to duplicate the exit pupil with a grating or diffuser [44]. This method implies that a real image somewhere before the final eyepiece is being diffused. Initial experiments show promise for inclusive systems but the real image for an augmented vision display may not be accessible.

9.6 Viewing Optics

A more compact optical design for the viewing optics may be necessary for a head mounted display. In the current design, the location of the scanner below the eye may disrupt downward vision. An approach where the scan comes into a partially reflective element from the side of the head would be more desirable for an operational head mounted display. However, any system incorporating the scan from the side of the head would probably be an off-axis optical system. Off-axis optical systems (as opposed to on-axis systems used in this project) inherently contain more complicated aberration and optical problems [3]. The final reflective element may be partially reflective plastic or perhaps a holographic combiner. The most desirable configuration for a head mounted version of the system would resemble eyeglasses more than it would the traditional helmet type head mounted displays.

9.7 Variable Accommodation

One problem in current helmet mounted display technology is the conflict between vergence and accommodation cues. Both cues assist in telling the brain the depth of an object. The vergence cue is related to the angle between the two eyes - the closer the object the greater the angle. The accommodation cue is related to where the eyes are focused - the closer the object the shorter the focal length of the eye. In current helmet mounted displays, the focal length must remain fixed because the screen is focused to a constant distance. Vergence continually changes, however, in stereoscopic helmet mounted displays. The discrepancy between the vergence and accommodation cues may potentially cause a user to feel somewhat ill [47].

The Virtual Retinal Display has a unique potential to create picture element specific wavefronts entering the eye. The VRD generates each pixel at different points in time and it is therefore possible to modulate the light wavefront for each picture element. The challenge for the implementation of this potential is to devise a variable focal length lens with switching speeds on the order of the pixel modulation switching speed.

9.8 Size and Weight

The eventual size and weight of a head mounted version of the display should be minimized. The design and construction of a head mounted display involves optical, mechanical, electrical, and industrial design. Furthermore, the final decision of size weight (and cost) should depend upon specific applications. Some applications have stricter tolerances on size and weight. Also the additional weight of the display produces a rotational moment which may contribute to fatigue. It matters not only how much the display weighs but also where the weight is in relation to the center of inertia for the head's muscular and skeletal system. Ideally for longer duration applications, however, the display should weigh less than 16 [oz.].

9.9 Summary

Remaining research relates primarily to system size and system performance. Smaller light sources are needed to make the display portable. Light emitting diodes or directly modulated laser diodes show the best potential to solve the problem. The exit pupil size is a serious and challenging problem for this device. The exit pupil problem must be solved to allow the user to gimbal the eye while looking around in the display. The Mechanical Resonant Scanner should incorporate phase detection in the future. Sinusoidal correction in modulation is also needed to compensate for the sinusoidal scan of the MRS.

The demonstration of a full color (RGB), augmented vision (see-through), Virtual Retinal Display represents a major step in pioneering efforts to create high performance, affordable, visually augmented displays. The display technology in this thesis, the Virtual Retinal Display, brings us closer to practical applications of virtual reality and augmented vision in fields such as medicine, manufacturing, and personal communications where overlaying computer information over the real world can enhance performance.

Human Interface Technology Laboratory