|Wearable Low Vision Aid's papers
Wearable Low Vision Aid main page
Wearable Low Vision Aid
Persons that are visually impaired have great difficulty navigating and avoiding obstacles as they walk even when using a cane or seeing eye dog and especially under low light levels.
Creating a portable, low-cost, assistive device to aid the visually impaired is the goal of the NSF-sponsored Wearable Low-Vision Aid (WLVA) project. The prototype WLVA uses machine vision to identify walking hazards and a see-through head-mounted scanning fiber display to present icons indicating the location of potential hazards. The scanning fiber display projects laser light through a vibrating optical fiber in order to project an image onto the retina. In this report we describe the engineering of a low-cost portable WLVA that incorporates infrared (IR) illumination and efficient machine vision algorithms to identify potential walking hazards and a scanning fiber display to present bright icons to warn the user.
The prototype WLVA consists of three major components: a head mounted display (HMD), backpack mounted equipment, and software. The HMD incorporates the scanning fiber display and optics mounted in a tube on one side of a spectacle frame, and a video camera with IR light emitting diodes mounted on the other side. The backpack-mounted equipment consists of a laptop computer, an embedded processor, and hardware to drive the scanning fiber display.
The software includes a machine vision program run on the laptop computer to identify potential collisions, an embedded processor and program to control the scanning fiber display, and a graphical user interface to facilitate setting parameters for the embedded processor and generating easily recognizable icons.
The main components (described below) of the scanning fiber display, video camera with IR LEDs, and brightness control knob were not designed for low weight, creating a 470g spectacle frame including attachments.
The scanning fiber display consists of two bimorph piezoelectric actuators, the optical fiber, and lenses. The small core optical fiber is attached to the end of the fast scan piezo, which is coupled orthogonally to the slow scan piezo. The slow scan piezo is cut to vibrate at its first mode of resonant vibration. The fast scan piezo and slow scan piezo are driven independently at 3 kHz and 60 Hz, respectively, to create a raster scan pattern. The other end of the optical fiber is connected to a laser diode, which modulates the light intensity synchronously with the fiber position in order to create multiple low-resolution icons.
The current display creates a scanned object plane of approximately 4mm×3mm with 100×28 separated pixels, displayed at 30 frames per second. The scanning fiber display is constructed by hand using components that cost less than one dollar. The bimorph piezos are cut to an appropriate length and width using a rotary cutter, and glued together. The optical fiber is chemically etched to reduce the fiber diameter and improve scanning dynamics. The optical fiber is glued to the tip of the fast scan piezo. The vibrating end of the fiber is trimmed with a CO2 laser in order to achieve maximum deflection at the first mode of vibratory resonance for the optical fiber cantilever.
The scanning fiber display creates an audible hum at the resonant frequency of the fast scan piezo (3kHz). Mounting a second piezo next to the fast scan piezo and vibrating it at the same frequency, but out of phase, results in an interference pattern that significantly attenuates the hum. Preliminary tests have recorded a 14 dB decrease in noise by using this method.
A 1.5 inch Delrin tube supports the fiber scanner and allows adjustment of the lenses. The scanned laser image is reflected onto the user's retina by a small mirror (or beam-splitter for optical see-through mode) mounted to the end of the tube. By allowing the WLVA user to see their surroundings at all times, situational awareness is maintained. The small mirror inset or see-through beam-splitter design allows the displayed icon to augment the user's vision. The high-brightness of the laser diode or future LED source make this display suitable for use in outdoor conditions with these various display modes.
A color video camera with a ring of 24 IR LEDs is mounted on the right side of the HMD. An optical filter is mounted in front of the camera lens to block visible light. Custom circuitry synchronizes illumination of the IR LEDs with alternating video frames. The video is captured and processed in real time by the laptop computer to identify and locate potential hazards up to 12 feet away. The head mounted video camera is angled down slightly to capture hazards from ground level up to head level.
A knob is located behind the camera that allows the user to reduce the brightness of the display by adjusting power to the light source for darker indoor or nighttime use. Currently, the display brightness is set for upcoming hazard avoidance testing in indoor lighting conditions according to our approved human subjects testing protocol involving low vision volunteers.
The backpack mounted equipment includes a laptop computer and an aluminum case containing control hardware and batteries. The total weight of the backpack equipment is 4.5 kg.
The laptop computer is a Dell Latitude with a 1.8 GHz processor. A video capture card (Dazzle Digital Video Creator 80) captures video at a rate of 30 frames per second. The laptop communicates with the embedded processor through the serial port.
Custom hardware was developed to control the scanning fiber display so the laptop computer could be dedicated to time-critical machine-vision hazard detection algorithms. The hardware used to control the scanning fiber display consists of an embedded processor, a first-in first-out (FIFO) frame buffer, and other discrete components. An Atmel ATMEGA128 generates the frequencies to drive the piezos, the synchronization signals to keep the frame aligned, and handles communication with the laptop and the FIFO frame buffer.
The scanning fiber display generates binary (uniform brightness) images but could easily be adapted to display gray-scale images. Static icons (a single frame projected repeatedly) or dynamic icons (a sequence of several frames) can be displayed. The FIFO frame buffer (IDT 7208) facilitates storage and retransmission of the pixel data (icons or text). The pixel clock frequency (laser modulation frequency) is divided down by the ATMEGA 128 programmable counters to generate the horizontal scan frequency and the vertical scan frequency. This method keeps all of the clock signals in phase and eliminates the phase locked loop used in the original bench-top prototype. The ATMEGA 128 programmable counters also generate horizontal and vertical synchronization pulses that are used to adjust the frame alignment. A ThorLabs (LPS-3224-635) 3mW pigtailed red laser diode (633nm wavelength) is the light source. For safety reasons the laser diode is driven at only 5% of its rated power during development creating an intensity of 230uW/cm2. A reading performance study in the see-through mode showed blue (458 nm) light may be easier for low vision subjects to see in the augmented display mode. The light source could be changed to a blue laser diode at additional cost. LED's brightness has increased rapidly in recent years, making pigtailed LEDs the preferred light source in the future.
The WLVA software includes machine vision programs running on the laptop computer and a scanning fiber display control program running on the embedded processor with a GUI that facilitates initial setup of the embedded processor.
An alternating IR-flash video capture technique is used to discern between objects in the foreground and objects in the background. Two consecutive frames are captured; one illuminated by IR LEDs and the other not illuminated by IR LEDs. The non-illuminated frame is subtracted from the illuminated frame to generate a differential measurement of luminance. This differential frame represents the light returning to the camera from the IR LEDs. Objects in the foreground reflect more light than objects in the background; therefore close objects appear brighter in the differential image. Bright areas are analyzed by their average luminance and position. The characteristics of these bright images are tracked over several frames to determine if they are growing in size and are therefore likely to present a collision hazard. The location of an imminent collision hazard determines which icon and where it is displayed in the HMD.
The embedded software facilitates loading icons into the FIFO frame buffer, selects which icon to display, and controls the parameters for adjusting the scanning fiber display. During development or user customization, icons are transferred from the laptop computer to the embedded processor through a serial link, and then to the FIFO frame buffer for projection. The embedded software adjusts the frame alignment by changing the timing of data output by the FIFO frame buffer. After the customization is complete, several sets of icons are stored in the non-volatile program memory of the embedded processor. Several different hazard icons can be customized to the user's preference and environment. The icons can be displayed in various locations in the visual field to warn the user of the location and proximity of potential walking hazards.
A Matlab GUI is used to communicate with the embedded processor during initial setup. Icons can be developed quickly on the laptop computer and saved as bitmap files. The setup GUI can import the bitmap icons, group the icons, and transmit the icons to the embedded processor. The setup GUI permits rapid changing of the parameters for the embedded processors without the need to recompile and download the embedded software. The icons can be designed to flash or increase in size to indicate increasing probability of collision.
The WLVA prototype is the first system to use a scanning fiber display in a wearable low vision aid. The IR flash illumination provides enhanced functionality at night, efficient hazard detection capability, and early warning for a broad range of hazards. The scanning fiber display's high brightness, lightweight, see-through design, and low cost make it an ideal choice for this application. The scanning fiber with laser diode source makes it possible to augment a person's vision without obscuring their natural visual ability. Although the icons have a low pixel count, the user can pre-select the icons for specific warnings making recognition rapid. This enhancement of the visual system is preferred to using an auditory warning that could obscure natural auditory cues that are critical to the visually impaired.
The WLVA system is designed for low cost and versatility. The monocular near-eye display can be added to a pair of spectacle lenses with fiber-scanning display components costing less than one dollar. Therefore the augmented display can be considered disposable eliminating the need for repair. The laptop computer is commercial off-the shelf technology that can be used in fixed locations like at school, work, and home when not being used for machine vision hazard detection.
The laptop computer could be interfaced with a GPS unit to display navigational aids or an optical character reader for text-to-speech conversion.
Testing of the WLVA system and its hazard detection and identification capabilities is expected to begin in early 2004 at the University of Washington with low vision volunteers.
We have prepared illustrations of how the WLVA might help the user to avoid a large pot and a bicycle rack. There is also a first-person perspective of avoiding the pot. These illustrations are also available in Windows Media format: the pot, the bike rack, and the first-person view of the pot.
An illustration of a former, bulkier design is also available.
Two graduate students received their MS in mechanical engineering and five undergraduates participated in this research project, most all being first authors of journal or conference papers. All students obtain the unique experience of designing and testing the assistive devices with our HITLab staff, one of whom is legally blind.
Subject to continued funding, the WLVA hardware will undergo a significant reduction in size, using a single tubular piezoelectric actuator less than 2mm in diameter to generate over 50 times more pixels while maintaining its extreme low cost. A printed circuit board has been designed to significantly reduce the weight and size of the backpack electronics as well.
This research was funded by NSF Research Grant to Eric Seibel #9978888, Research to Aid Persons with Disabilities, with REU supplements allowing undergraduate students to contribute significantly.
Eric Seibel <eseibelhitl.washington.edu>