In the spring of 1994, King and Suzanne Weghorst, HITL director of interface development, conducted a study to examine facial expressions during time sensitive, goal-oriented tasks, namely the Towering Inferno scenario. These tasks involved requesting multimedia information to aid in planning and executing an escape (the goal). Delays were introduced in the fulfillment of information requests to simulate network activity. This scenario was chosen because it fulfills the needs of the AFOSR work with HITL. It also simulates a wide variety of advanced computer-based tasks, such as services on an information super-highway. Cognitive load was manipulated by the presence or absence of an on-screen map. The user's subjective reaction to the length of the delays was measured using a metric developed by Weghorst. Facial expressions were recorded and time synchronized with various commands and requests, and with the machine events. This allows the expressions to be directly correlated with machine state via a time series analysis. In this way, King hopes to determine the characteristic situations in which facial expressions are exhibited.
The preliminary results show a higher rate of expressions when the map is present. Asymmetric facial expressions, expressions which occur only on one side of the face, have been found which have interesting implications for hemispheric dominance during human computer interaction. King has also found the presence of expressions which are not emotional. He hypothesizes that these are cognitive facial expressions which human observers might label as perplexity or confusion. These sorts of expressions are certainly important in the primarily cognitive virtual environment.
King is also developing cognitive models particularly of emotion for human computer interaction. These models may some day serve as part of the user model for semi-intelligent interface agents which act based on machine state and facial information. King has defined the problems with the existing models of emotion which will have to be addressed in their application to human computer interaction. A major shortcoming is the existing model's dependence on a person's reaction to physical phenomena (agents, events and objects). He theorizes that there are basic qualitative differences between physical and virtual phenomena which must be taken into account when designing new models.
This summer, King conducted research at the Advanced Telecommunications Research Institute (ATR) in Japan to investigate user's appraisal of virtual phenomena. He is exploring the classification of phenomena as combinations of the three traditional distinctions. He is studying the appraisal of "agents" in particular. The users are asked to not only appraise the phenomena but rate the level of intelligence that it possesses or might possibly possess. Their facial expressions are also recorded during the presentations and correlated with the user's appraisals of the phenomena and its intelligence.
Since interface agents are increasingly a topic of research and application, this study may have direct benefits to the design of the physical representation of an agent to the user. For instance, the designer may be able to use facial expression exhibitions to determine how much intelligence a user prescribes to an agent or whether he/she even thinks of the representation as an agent. In any case, the results should further the development of an emotion state model and allow us to better understand the virtual environment.
Finally, King hopes to integrate gaze and pupil size into his research this winter as HITL plans to acquire an eye tracking and pupilometer system. This equipment will allow for a holistic model of the face in which all of the facial nonverbal behaviors a re studied en suite. Evolutionary factors have combined to place all of these behaviors in a very limited area. They provide much information in themselves, but a gestalt may provide more information than the three do separately. For instance, pupil size is almost unusable by itself since it has many physiological correlates, but it may be very useful as an arousal and cognitive load measure when combined with facial expressions and gaze. King believes this holistic approach will be necessary in finally realizing the human computer dyad.