click to view contact details click to search using Google SiteSearch click to view directions to the lab
Expert Surgical Assistant's papers
Expert Surgical Assistant main page

where we are
contact us
research projects
news & events
hitlab nz
hitlab au




Expert Surgical Assistant

Mark Billinghurst has extended his work with HIT Lab PhD graduate Jesus Savage-Carmona on intelligent multi-modal environments to encompass an intriguing medical application. Working with HIT Lab research engineer Peter Oppenheimer and Lab associate Dr. Chuck Edmond, an otolaryngologist at Madigan Army Medical Center, Billinghurst has demonstrated a prototype expert surgical assistant for simulation of a common paranasal sinus surgery procedure.

The system, which incorporates knowledge of the endoscopic procedure into a structured rule base, interprets the user's multi-modal inputs (currently voice and virtual endoscope position) and interacts with the user dyadically. While performing the simulated procedure, the user can query the system about anatomy and the specifics of the procedure, asking the system to identify features or demonstrate maneuvers. In turn the system recognizes the user's actions, and can provide vocal and visual feedback, as well as warnings when the user is about to execute a dangerous maneuver.

In addition to enhancing simulation for training purposes, the expert surgical assistant can also be used for guidance during actual surgical procedures. This assistance can be particularly helpful for procedures in which situation awareness is diminished due to the complexity of the anatomy or to surgical complications. Billinghurst and his colleagues plan to extend the rule base for procedures in this anatomical region, and to build a toolkit which will permit physicians to easily construct expert assistants for other procedures using other simulated anatomy.


Mark Billinghurst <mark.billinghurst at>