Viewing Ocular Tissues with A Stereoscopic Endoscope
Coupled to a Head Mounted Display (HMD)
Greg Heacock (1)
John Marshall, Ph.D.(1),
Prof. Franz Fankhauser, M.D.(2).
Toni C. Emerson, MSLIS (3)
(1) University of London, UMDS, St. Thomas's, U.K.
(2) University of Bern, Switzerland
(3) HIT Lab, University of Washington
In surgical procedures outside ophthalmic practice, minimally invasive surgical techniques are becoming more prevalent due to factors of improved patient recovery, reduction of postoperative pain, and cost savings. These techniques have not been widely accepted in ophthalmology, since transpupillary viewing provides adequate visualization in most cases. In cases where fundus viewing is not possible, such as those with a cataract or hemmoragic viterous, micro minimally invasive surgical techniques could provide a means to acquire a view. Before these techniques find wide use in ophthalmology several problems must be overcome. The problems are due in part to five issues encountered when integrating a video monitor with a surgical procedure; the remote location and orientation of the monitor, loss of hand-eye coordination, magnification, field of view, and the lack of depth of field and loss of stereopsis. A new method of viewing the fundus stereoscopically has been developed to address these problems. The stereoscopic display is worn on the head of the surgeon, and is interfaced to a pair of videoendoscopes inserted into the patient's eye. The design of the head mounted display allows the surgeon to glance up to see a three dimensional display of the fundus, or to look down to observe the instrument entry site of the eye.
Key Words: Head Mounted Display (HMD), Stereoscopic Display, Videoendoscope, Remote Viewing, Liquid Crystal Display (LCD), Ray Trace, Charge Coupled Device (CCD), Minimally Evasive Surgery (MIS).
In 1983 when the first videoendoscopic systems became available, technological deficiencies and behavioral patterns limited their use. The resistance to acceptance was due to lack of technique experience, resolution limitations of the charge coupled device (CCD) cameras, and poor color reproduction. Now, ten years later, the move to minimally evasive surgery (MIS) for cost cutting and patient benefit is realized as component technology has improved with high density fibers and gradient index lenses. Because of the relative ease of direct viewing of ocular tissues with transpupillary visualization techniques, ophthalmology was at the forefront of imaging science for many years. With the advent of magnetic resonant imaging and other techniques, however, other medical disciplines have adopted these techniques and become more sophisticated in the field of imaging and image analysis. In the more challenging cases, particular problems of viewing deep ocular tissues are often associated with disease processes which inhibit normal transpupillary viewing methods. Therefore in cases where a corneal opacity, cataractious lens, or hemmoragic viterous exclude transpupillary viewing, the techniques MIS would provide a means for observation. When considering the design factors of an ophthalmic MIS system, much can be learned from other surgical disciplines where these procedures are common, and the difficulties are known.
The first problem encountered in MIS techniques in general surgery are the constraints imposed upon the manipulation of the remote surgical scene through a small surface incision. In general, such incisions rarely exceed 10 mm, as this is sufficient to allow passage of the optical viewing system and surgical manipulator. In ophthalmic MIS, a 10mm incision would be too large. Even greater constraints exist on transcleral instruments, and in such techniques as vitrectomy, the normal instrumentation passes through a 10 gage needle opening. Therefore, the videoendoscope must be similar to other ophthalmic surgical instruments in their ability to pass through small openings such as a 10 gage needle.
Other problems in MIS are visual and can be described as; the limited size of the field of view, the complete lack of depth perception, and loss of image orientation offered by a video image on a monitor. Technique learning difficulties which have been previously described, 1, discuss the problems encountered by surgeons who must overcome loss of depth perception, video image to tissue axis inconsistency, and the remote placement of the video monitor. The length of learning curve is another important factor inhibiting the acceptance of the endoscopic procedures, 2,3. To master the skills required to perform endoscopic procedures, a surgeon must reprogram their natural skills coupling visualization with handmovements to this more complex interface. These factors result in large differences in procedural durations and outcome comparing inexperienced to experience surgeons. For example, the operating time for surgeons performing hemicolectomys decrease as experience was gained, with times ranging from 330 to 120 minutes, 4. Further, the complication rate for laparoscopic procedures may be four times higher in the first 100 procedures performed by a surgeon, 3.
Regardless of the difficulties of endoscopic surgery, once perfected, the advantages to the patient are considerable. Less tissue trauma reduces hospitalization time and enhances recovery. Secondarily, cosmetic considerations are improved due to the small incision site required for instrument entry. Given the above problems the role of such devices in ophthalmology may seem even more limited, however, they could confer important advantages in certain circumstances. Where transpupillary observation methods may not be possible, a videoendoscope may provide a view of important structures. Another important advantage of videoendoscopy is the ability of many viewers to observe a procedure from the surgeons perspective. Reports, 5, describe how teamwork is enhanced by allowing all members of the surgical team to view a procedure, facilitating instantaneous interventions, and consultations. For teaching, any number of students may observe a procedure, since the image is may be displayed on monitors.
To solve the visual problems of videoendosopy, and realize the important benefits of MIS and remote viewing techniques as applied to ophthalmic surgical procedures, we must understand the problems associated with video displays in medicine, 6. These problems may be divided into five issues; the remote location and orientation of the monitor, loss of hand-eye coordination, magnification, field of view, and the lack of depth of field and loss of stereopsis.
The remote location of the monitor and its orientation in the surgical setting have an effect on both the doctor's perception of the surgical field, and the difficulties encountered when interacting with it. Much of the difficulty encountered by a surgeon when learning the techniques of videoendoscopy, relate to the disconnection between instrument movements on the video monitor, and their hand movements. With the displacement and axial rotation of the monitor, the doctor must relearn the image to tissue relationship at each procedure, and with each manipulation of the camera. With this decoupled image, the hand-eye coordination of the doctor suffers. For example, with a 90 degree rotation of the camera, an intended movement South to North direction, will result in a display movement in an East to West direction, presenting the doctor with a working field that is difficult to comprehend.
Magnification and field of view affect the perception of the doctor viewing the monitor image. With a given field of, view the surgeon requires enough magnification to perform the required task, however, excessive magnification is confusing because manual movements within the surgical scene will be translated to extremely large, rapid movements on the video monitor. This problem can be further exacerbated by a small field of view, because if the instrument tip moves outside of the view, the clinician may have no concept of the location, and in turn may cause tissue damage with hunting motions while trying to reestablish orientation.
A surgeons ability to locate their instruments relative to the target tissue is due to the stereo image clues perceived from the convergence angle of their eyes, and the differences between the images in their left and right eyes, 7. The single screen video monitor that the surgeon uses as the image source, by design, has no stereo information. The surgeon can only accurately place the video monitor in space, the location of the original image is unknown. When a doctor moves their instruments in the operating field, the resultant image on the monitor contains no spacial information on instrument location, and no concept of depth of field. It can be observed in the oparatory during a MIS procedure, that the surgeon often makes small false moves with his instruments to establish the surgical field orientation, prior to a true move required by the procedure.
Having discussed the nature of the main problems confronting videoendoscopy, there is a secondary set of considerations that should be reviewed before postulating practical solutions. In any given surgical task the importance of the visual elements need to be understood. With the videoendoscopy, sufficient light, and accurate CCD's must be available to insure true color representation. During operations such as papillotomy, the surgeon must be able to distinguish the color difference between blood and bile for example, 5. Improper color projection or misunderstood image processing would be unacceptable. With modern designs of videoendoscopes, these problems of color imaging are reduced, however, illumination of the surgical field is still a critical factor.
There are parallels between video surgery, and video information display found in modern fighter aircraft. The amount of information that needs to be assimilated and understood by pilots and surgeons alike is staggering. In the design of complex fighter aircraft cockpits, the need to supply the pilot with information that is perceived and processed in the natural way that people interact with their world was a major consideration, 8. For several years pilots in high performance aircraft have used "head up" display systems, with images being projected onto cockpit canopies in an effort to simplify their task of controlling the aircraft, identifying targets, maintaining communications, and defending themselves. Accommodation errors and other problems resulted in the concept of a head mounted display (HMD) system, 9.
As with the solution of the pilots problem of mass data assimilation with the use of HMD's, in the surgical environment such systems would certainly resolve the problem of remotely located monitors. By correct alignment of the video image relative to the surgeon's hand and instrument position, they could also contribute significantly to reducing problems of hand-eye coordination. To further optimize this image, two other considerations should be investigated. First the brightness of the video image must be balanced with that of the real world. Secondly, its spacial location should consider the adaptation and accommodation of the viewer, 10.
However, before integrating any HMD into the surgical setting, real benefits must be demonstrated to the doctor. Paramount amongst these is the need to display information in a format that is understandable by the viewer, is easily assimilated, relates to the real tissue, and is contained in an unobtrusive head mount. There are many devices commonly worn on the head by an ophthalmic surgeon, such as; binocular indirect microscopes, specticle mounted converging telescopes, and laser delivery systems. For many surgeons, operating schedules and complex procedures require them to be in the O.R. for many hours. Therefore, a surgical HMD, should be as unobtrusive as is possible. It should be light in weight, and balanced, allowing easy head movement without problems of inertia or neck strain. The HMD should also offer a clear visual path to the external world. Components that occlude stereo vision of the operation site, or optical surfaces capable of fogging due to perspiration, could be a hindrance during a critical procedure.
Recognizing the observation problems encountered by ophthalmic surgeons, the visualization possibilities of videoendoscopy, and the requirements of HMD's, we have developed a new method of capturing stereoscopic video information of internal ocular structures, and displaying it via a surgical HMD.
The integration of stereo video images of a surgical site, into a three dimensional (3D) video display is accomplished by capturing a real time left and right stereo pair of images with a videoendoscope pair, and then projecting this image pair to the surgeon via a HMD. First, the videoendoscope design will be reviewed, then that of the HMD.
fig. 1
Fig. 1 shows one of the videoendoscopes. In prototype form it consists of an imaging fiber 130mm in length and 2mm in diameter, a focusing lens system, and a micro charge coupled device (CCD) with a pixel array of 510 horizontal X 492 vertical. There are no empirical reasons that the fiber may not be reduced in size to 50 microns in diameter. The unit requires 12 V DC power, and produces NTSC standard video output. Two endoscopes are required to capture a stereoscopic image. The two endoscopes were clamped together with a six degree convergence angle as more accurate manipulation tasks require some image overlap, 11. This is a similar convergence angle to that of the surgeon's eyes when observing close objects in their near field.
This assembly was inserted through a scleral incision into a model eye, (MIRA). This model eye is fabricated from a soft plastic material, and has a painted fundus with detail such as; a disk, macula, and retinal vessels. To provide for a more realistic optical model, the eye was filled with water for the imaging experiment. In fig. 2 the eye model is seen with the stereo videoendoscope inserted through the sclera.
fig. 2
The surgeon wearing the HMD sees the left and right stereo image pair projected into their respective left and right eyes. To perform this function, the HMD must manage the human interface functions of; alignment of the displays, accommodation adjustment, video image location, controlling distortion of the display across the image plane, and maintenance of equal contrast and brightness between both video images. To manage these factors, the HMD required 6 major components as shown in fig. 3. The components of the system are; the housing, head strap, liquid crystal display (LCD) screens, aspheric lenses, focusing rings, and video cable.
fig 3
The housing holds the LCD's in a fixed position relative to each other to reduce the possibility of image misalignment, and at a sufficient spacing from the surgeons eyes to alloy spectacles to be worn.. The focusing ring moves the aspheric lens closer of further relative to the LCD, to facilitate focusing of the image, and to adjust for accommodation variances between different users. Additionally, by changing the LCD to aspheric lens spacing, it is possible for the surgeon to adjust the location of the virtual image formed by the HMD. This important function allows the surgeon to place the virtual image on his hands, reducing the need to accommodate to varying image planes, and providing a coupled image to instrument working field.
Fig.4 shows a raytrace of the spatial relationship of the surgeons eye, the aspheric lens, and the LCD display.
fig 4
The aspheric lens is composed of two convex surfaces each having an aspheric shape. This focuses the image from the 13.5mm X 10mm active matrix LCD screen located approximately 75 mm in front of the viewers eye, onto the fundus of the viewer's eye with minimal distortions. The perceived field of view is approximately 35 degrees horizontal, and 25 degrees vertical.
A critical adjustment was found to be necessary during the pilot study. We found that perception of the stereo images was best when the contrast and brightness of the two LCD screens was carefully adjusted to be equal. This was accomplished by sending a mono video image to both HMD LCD's, and adjusting the voltage potentiometer controlling the brightness of the respective left and right LCD backlight to equivalence.
The video output was taken from each CCD videoendoscope and connected to the respective left and right HMD video cable. This system, produces the stereo image pair shown in fig 5, A B, left and right respectively. With an image display rate of 60 hz.
fig 5
It is important to connect the right and left video to the respective display electronics. Incorrect connection results in a very confusing display, as the normal area of binocular overlap is located on the left and right edges of the respective screens, and not centrally.
The HMD is worn by the surgeon as shown in fig.6, and the 3D image is adjusted to be coincident with the preferred work place of the surgeon by moving the focus ring.
fig 6
Next the CCD cameras are rotated to provide the surgeon with an image pair that is erect relative to the viewing position. In this way, the surgeon is able to view the instruments or the 3D image of the eye's interior, by slightly adjusting his gaze upward or downward. The view seen by the surgeon is depicted in fig.7. The internal structure of the eye is located in the upper half of the surgeon's field of view, and a unencumbered view of the sclurotomy and or surgeons hands are visible in the lower half.
fig 7
We have demonstrated a device that enables stereoscopic video images to be obtained from within the eye. In realizing our prototype system we have overcome all of the implicit technical problems. Including the visual factors of remote location and orientation of the monitor, loss of hand- eye coordination, magnification, field of view, and the lack of depth of field and loss of stereopsis. Cost considerations prohibited the use of micro fibers in this initial system, however, pilot experiments coupling them with the current CCD have shown that there seem to be no technical problems in this advancement.
We realize that penetrating the globe with an endoscopic fiber is not a diagnostic procedure that will be utilized in the absence of surgical intervention. However, we feel that this device may have a number of roles in surgery, visual science and teaching. As previously indicated, such devices may have a role in vitrectomy, particularly where structures are occluded from direct observation or when they have complex structure or geometry. In the latter case, high magnification 3D images may facilitate surgical dissection. The ability for several individual to view such structure prior to dissection may enhance patient care. It would certainly facilitate close support from surgical assistants. In the laboratory, devices of this type would enable realtime analysis of movements of cillary muscles during accommodation as the normal observation methods of transpupillary viewing and scleral depression deform the interesting tissues too much. In both the operating theater and the laboratory, the possibility for recording a complete procedure, and playing it back to any number of students would improve the training of individual on procedures such as the placement of IOL hapitcs.
The device is now being deployed amongst a number of vitero-retinal surgeons to ascertain the needed improvements so we can input a body of clinical experience in further iterations of this device.
This work is in part a contribution for a Ph.D. for Greg Heacock, at the University of London, UMDS, St.Thomas's.
1. Barrett D, Green R, Copeland S: Arthroscopic and endoscopic skills: a method of assessment. Annals of the Royal College of Surgeons of England 73:100-104, 1991
2. O'Rourke N, Heald R: Laparoscopic surgery for colorectal cancer. Br. J. Surgery 80:1229-1230, 1993
3. Capelouto C, Kavoussi L: Complications of laparoscopic surgery. Urology 42:2-12, 1993
4. Monson J, Darzi A, Carey P, Guillou P: Prospective evaluation of laparoscopic-assisted colectomy in an unselected group of patients. Lanset 340:831-833, 1992
5. Knyrim K, Seidlitz H, Vakil N: Current developments in electronic endoscopy. Endoscopy 24:Suppl. 2:502-505, 1992
6. Heacock G, Marshall J: The use of non-immersive virtual reality in ophthalmic diagnosis and its potential for laser surgery. Lasers and Light in Ophthalmology 6:No.3:135-141, 1994.
7. Ikehara C, Cole R: Visual-motor correspondence in stereoscopic video displays for teleoperated manipulator tasks. SPIE Proceedings, Stereoscopic Displays and Applications IV, 1993.
8. Furness T: Harnessing virtual space. SID Proceedings 1988
9. Furness T, Kocian D: Putting humans into virtual space. Armstrong Aerospace Medical Research Laboratory, Wright Patterson AFB, Ohio 45433
10. Iavecchia J, Iavecchia H, Illiana R: Eye accommodation to head-up virtual images. Human Factors Society Dec. 1988