The Use of Sketch Maps to Measure

Cognitive Maps of Virtual Environments.

Mark Billinghurst and Suzanne Weghorst

Human Interface Technology Laboratory

FJ-15, University of Washington,

Seattle, WA 98195.

+1-206-616-1493

{grof,weghorst}@hitl.washington.edu

ABSTRACT

Cognitive maps are mental models of the relative locations and attributes of phenomena in spatial environments. Understanding how people form cognitive maps of virtual environments is vital to effective virtual world design. Unfortunately, such an understanding is hampered by the difficulty of cognitive map measurement. The present study tests the validity of using sketch maps to examine aspects of virtual world cognitive maps. We predict that subjects who report feeling oriented within the virtual world will produce better sketch maps and so sketch map accuracy can be used as an external measure of subject orientation and world knowledge. Results show a high positive correlation between subjective ratings of orientation, world knowledge and sketch map accuracy, supporting our hypothesis that sketch maps provide a valid measure of internal cognitive maps of virtual environments. Results across different worlds also suggest that sketch maps can be used to find an absolute measure for goodness of world design.

KEYWORDS Cognitive Mapping, Virtual Environments, Sketch Maps, Mental Models.

INTRODUCTION

Whether in real or virtual space we form cognitive maps to deal with and process the information contained in the surrounding environment. Cognitive mapping is formally defined by Downs and Stea [6] as:

"..a process composed of a series of psychological transformations by which an individual acquires, codes, stores, recalls, and decodes information about the relative locations and attributes of phenomena in their everyday spatial environment."

An individual's cognitive map is an active information seeking structure of which spatial imagery is but one aspect [14]. Cognitive maps are also made up of memories of objects and kinesthetic, visual and auditory cues [8].

The fundamental importance of an effective cognitive map is that it allows two questions to be answered quickly and efficiently: Where is that? How do I get to there from here? Thus human spatial behavior relies upon and is determined by the individual's cognitive map of the surrounding environment. In addition, the perception of the environment itself is always guided by some sort of cognitive map, so an inaccurate or incomplete cognitive map leads to disorientation and confusion[l4].

Designing virtual worlds through which subjects can navigate and orientate themselves successfully requires an understanding of cognitive map formation in virtual environments. Considerable research which might be brought to bear on this topic has been conducted on the development of cognitive maps and how they affect real world behavior.

In exploring how people formed mental images of a city Briggs[4] has identified three complementary ways in which cognitive maps are created:


Of these, an individual's sensory modalities provide direct sources of information and are more effective in cognitive map formation than indirect sources[6].

Cognitive maps are created as the result of active and passive modes of information processing [14]. Generally, active information processing gives the greatest meaning to the information processed and produces more information for the moving perceiver. Thus the information produced by locomotion is fundamental to an individual's spatial orientation.

An individual's cognition of the environment is not only a function of the behavior by which information is obtained but also depends on the characteristics of the environment [4]. The amount of information gained by each sensory modality is also environmentally dependent [16].

Aside from the way cognitive maps are formed, the types of information stored in a cognitive map are also of interest. Kuipers[10] suggests that a cognitive map consists of five different types of information, each with it's own representation: Topological, Metric, Route Descriptions, Fixed Features and Sensory Images. Different techniques are needed to measure each different information type.

Finally, Lynch[12] notes the uniquely personal nature of cognitive maps. Across different cultures he found that different groups may have widely different images of the same outer reality. Also, on an individual level, what an observer sees is based on a common exterior form, but how the observer interprets and organizes this form is unique. This interpretation governs how the observer directs his attention and this in turn affects what is seen. So at both a societal level and a cultural level cognitive maps are highly individualistic.

COGNITIVE MAPPING - THE VIRTUAL EXPERIENCE

As suggested above, cognitive maps are most effectively formed by active interaction with the environment using many different sensory modalities. However, in a virtual world there is typically sensory degradation and a lack of many of the perceptual cues used in the real world. Downs and Stea [6] point out that any filtering of information before it reaches the sensory modalities affects the cognitive map. This is the case for virtual environments. For example, the visual modality may suffer from low image resolution, poor image quality or a reduction of the peripheral field. In real environments, Alfano and Michel[1] have shown the reduction of peripheral vision impairs perception and visuomotor performance, both of which are essential for cognitive mapping ability. In addition there are rarely any tactile or olfactory cues and often only limited auditory feedback. The study presented here examines some of the factors influencing cognitive map construction given current immersive technology.

METHODS FOR ASSESSING COGNITIVE MAPS

One of the difficulties in studying cognitive mapping is the problem of extracting an external representation of an individual's internal map. By definition a cognitive map is highly subject-specific and, although individuals often record the same things in their cognitive maps, there is no evidence that they record them in the same way. Golledge[7] identifies four distinct methods for extracting environmental cognition information:

  • In our experiment we assess the subject's cognitive map through subject self-reporting and analysis of external representation.

    We are particularly interested in the subject's topological understanding of the virtual environment, i.e. knowing where they are and where everything else is, as compared with metric knowledge - knowing precise object location and distance between objects. Topological knowledge is generally more important than metric knowledge for effective navigation.

    A common approach for measuring topological knowledge was suggested by Lynch[12], who had subjects sketch maps to represent the mental models of their local cities. Lynch finds that sketch maps are more accurate when used for topological rather than metric analysis.

    Golledge[7] points out, however, that caution must be taken that sketch maps are not over analyzed. The disadvantages of sketch maps include trying to represent a three-dimensional cognitive map in two dimensions and the difficulties of quantitative analysis. They may also measure more than just spatial understanding of an environment, e.g. drawing or memory ability. Conversely, Blades[3] finds them reliable over time and Newcombe[15] comments that they are no less accurate than other cognitive techniques.

    Other common techniques used for cognitive map analysis include distance and angle estimation. However, Henry[9] found that distances were consistently underestimated in virtual environments and that angle estimation produced wildly varying results. Moreover, his subjects' sketch maps are topologically accurate even when the sketched distances are not. In a prior work we used a different technique for distance estimation and found similar results[l8].

    The present study is designed to asses the validity of sketch maps as a tool for measuring cognitive maps of virtual environments, particularly the topological knowledge of the cognitive maps. We predict that subjects who report feeling oriented and unconfused in the virtual world will later produce relatively accurate sketch maps, whereas subjects who report feeling disoriented and confused in the virtual world will produce less accurate sketch maps. In other words, if sketch maps are an accurate external representation of the subject's cognitive map then we would expect a correlation between the sketch map scores and subjective ratings of how oriented subjects felt within the virtual world.

    EXPERIMENT DESIGN

    Eighty four subjects experienced a number of simple virtual worlds and then produced maps. The worlds were constructed using Swivel and Body Electric software, and rendered on an SGI VGX. Participants wore VPL Eyephones and interacted with the virtual environment using a VPL Dataglove. Movement through the virtual environment was achieved by the users pointing in the desired direction they and making a "fly" gesture with the Dataglove. This movement was completely unconstrained so participants could be as close or far away from the world as they wanted. Collision detection was not used so participants could travel through objects.

    Each subject was initially trained on the same immersive virtual environment until they felt comfortable with moving and interacting within a virtual environment. Following this training, they were given a 24-question survey which asked for responses on a range of navigation, orientation, interaction, presence and interface questions. Survey responses were indicated on a 10-point anchored scale. These survey questions are reproduced in the appendix. Participants were also invited to comment about the experience in general.

    Table 1.0: The Different Characteristics of the Three Test Worlds.

    After the training session. subjects experienced one of three different virtual worlds for 10 minutes and were told to explore it as fully as they could. They were then asked to produce a map of the world that someone unfamiliar with the world could use to navigate around the world. The subjects also completed the same survey that was administered after the training world and were video taped for later observation of behavior patterns.

    If the sketch maps are an accurate external representation of the subjects cognitive map then we would expect a correlation between the sketch map scores and subject survey scores for orientation within the virtual world.

    World Differences

    Three different worlds were used to explore how differences in world design might affect the cognitive map formed and the resultant sketch maps. According to Darken and Silben's[5] world classification, each of them are "small", in that all of the world can be seen from a single viewpoint. They are also static, all their objects having positions and values which don't change over time. However, the density of each of the worlds varied considerably as detailed below. Each subject experienced only one of the test worlds.

    Virtual Valley

    Under Darken and Silben's scheme this is a dense world: it has a large number of objects and spatial cues; however, they are all placed in a logical manner. The world is bound on either side by tall mountain ranges that direct attention to the objects contained in the valley below. Objects within the world are all representative of what would be expected in a real valley and there are no hidden objects. Objects are clearly distinguishable by color and size, and there are a number of distinctive objects that could serve as landmarks. This world design would make it difficult for subjects to become disoriented.

    Cloudlands

    Cloudlands is a sparse world containing few objects. It contains a dominant ground plane with clusters of objects floating above it in cloud groups. One of these clouds contains a fish and star while the others are empty. The objects are incongruous and surprising - there is a floating cactus, stacks of multicolored planes, cones and small gray rocks. The are no environmental cues to direct attention other than the object clusters themselves. However, the sparcity of the world would also make it difficult for subjects to become disoriented.

    Neighborhood

    Neighborhood is a cluttered world containing clusters of buildings all closely grouped and each containing other objects. The buildings are largely the same size and color making it hard to distinguish between them, and the objects within them are almost all the same color as the buildings. The objects are all those that would be logically found in a neighborhood, such as trees, tables, glass and a piano but the similarity of the buildings makes it hard to precisely locate them. This world is generally confusing and disorientating.


    Table 1.0 summarizes the characteristics of the three test virtual worlds.

    SKETCH MAP ANALYSIS METHOD

    As mentioned before, one of the challenges of using sketch maps is analyzing the results. The maps produced are as individualistic as each of the cognitive maps of the subjects. Although sketch maps are commonly used in real world cognitive mapping there is no generally accepted method for their analysis. Useful approaches have been reported in Appleyard[2] Ladd[11], Moore[13], and Walsh et. al.[l7], among others; however these are used to analyze maps of large scale urban environments. Adapting these methods, we use a simple, purely topological technique. Each sketch map was given a set of goodness, object class and object positioning scores as detailed below:

    Map Goodness

    Maps were ranked for goodness on a scale of 1-3 by two researchers who were experienced in virtual environments but blind to subject identity and other correlated measures. The researchers were told to rank the maps on how useful they would be as a navigational tool if they were taken with them into the virtual environment. They were told to ignore the participants drawing ability and focus on how well the map represented the virtual world and the locations of the objects within it.

    Object Classes

    Each map was given a score according to the number of object classes present - for example, trees, rocks and mountains are each counted as separate classes. Using object classes is a way to assess completeness of a sketch map for a given virtual world.

    Relative Object Positioning

    To provide a measure of differences in cognitive maps for different worlds we scored maps according to relative object positioning. We used topological positioning and so scored objects if they were correctly positioned to the right or left, above or below, or clockwise or counterclockwise, depending on the specific world being represented. The specific object position was not important, only its position relative to other objects in the sketch map.

    Maps were given two positioning scores: a total object position score in which all the objects were scored, and a significant object position score where the five most commonly drawn objects for each world are scored. Relative object positioning is a way to assess the accuracy of sketch maps.

    RESULTS

    Although subjects were given no instructions on how to produce their maps, almost all of them drew three dimensional representations of the virtual world. This may be due to the small size of the worlds - sketch maps produced of large scale real world environments are usually two dimensional. Figure 1.0 shows typical sketch maps produced for the Cloudlands world.

    table 2
    Table 2.0: Goodness and Class Number correlations with virtual world orientation and knowledge across the test worlds.

    Within World Correlation

    If sketch maps can be used as an external measure of subjects' cognitive maps then there should be a strong correlation between map goodness scores and subject scores for the survey questions "Knowing where everything is in the Virtual World" and "Orientation in the Virtual World". To investigate this we correlated the object class and map goodness scores with the survey responses. Table 2.0 shows the correlation values of the map scores and survey scores. Although the map goodness rankings are highly subjective, the correlation between the scores given by the two researchers was very high; ( r = 0.86, 0.71, 0.70, for three worlds respectively, significant at p < 0.01).

    In the Virtual Valley and Neighborhood worlds object class and map goodness were both significantly correlated with the subjects' reported sense of orientation in the virtual world. For these two worlds, the map goodness score is also significantly correlated with subjects' knowledge of where everything is. However, this isn't the case with the Cloudlands world. The sparse nature of Cloudlands may make it difficult to produce an accurate sketch map. Cloudlands was also more three-dimensional that the other worlds with most objects placed high above the dominant ground plane, adding to the difficulty of producing a two-dimensional representation.

    Since cognitive maps are most effectively formed by active interaction with the environment, there should also be a relationship between map scores and the survey questions relating to interaction. This is indeed the case with Virtual Valley, where the map goodness rankings correlate significantly with the subjects survey score for ease of interaction (r = 0.882), ease of navigation (r = 0.865), ease of movement within the virtual world (r = 0.814) and ease of use of the Data Glove (r = 0.645). However, in the other two worlds the correlation between these survey questions and the map rankings were not significant.

    Between World Differences

    A two factor ANOVA was done on the survey results to identify world differences and possible gender-linked factors. There was a significant difference between worlds in subject's understanding of where everything was (F[2,22]=4.49, p < 0.025), and how oriented the subjects felt within each of the worlds (F[2,22]=3.314, p < 0.05). For both of these questions subjects rated Neighborhood world significantly lower than the two other worlds, as shown in figure 2.0. There was also a significant difference between the sense of dizziness reported by subjects, with those in Neighborhood registering the most dizziness, (F[2,22] = 3.95, p < 0.025). These results reflect the particularly disorienting nature of Neighborhood world.

    If sketch maps are representative of subjects virtual world cognitive maps, they should also reflect these world differences. The relative object position scores can be used to compare across worlds. For each world we defined the five most commonly drawn objects as "significant objects" and a relative positioning ratio was then calculated for each map:

    Ratio = Correctly placed significant objects divided by Total number of significant objects in map.

    An ANOVA revealed a statistically significant world difference for the significant object relative positioning ratios, (F[2,22] = 4.004, p < 0.025). A similar ratio was calculated for the relative positioning for all objects drawn in the sketch maps. In this case an ANOVA showed no significant world difference, (F[2,22] < 1.0 NS). figure 3.0 shows the relative positioning ratios for both sets of objects.

    figure 1
    Figure 1.0: Cloudlands world (upper left) and three typical sketch maps.


    Figure 2.0 Average subject orientation and world knowledge survey scores across the three test worlds

    In Virtual Valley over 90% of the significant objects that are placed are placed correctly, reflecting the well designed nature of the world. The difference in ratios from "significant" object placement to "all" object placement in Virtual Valley is largely due to a number of landmark objects which almost all of the subjects positioned correctly. The similarity of the "significantquot; and "all" object placement ratios in the other worlds may mean that there are fewer, if any, landmark objects.

    The difference in Virtual Valley and Neighborhood map scores correspond to the difference in subjects' orientation scores shown in figure 2.0. This suggests that "significant object- positioning scores may be used as a simple absolute measure of map accuracy and goodness of world design. It also implies that the sketch maps for these worlds accurately represents the topological knowledge stored in the subjects' cognitive maps.


    Figure 3.0: Average subject relative object positioning ratios across the three test worlds.

    CONCLUSIONS

    In this study we have investigated the applicability of sketch maps as an external representation of an individual's cognitive map of a virtual environment. We have found that sketch maps reflect differences both between worlds and within worlds.

    We used three methods to score the sketch maps, chosen for their simplicity and general applicability: map goodness and object class number for comparing maps from a given world, and the relative object positioning ratio for comparing maps across a range of worlds.

    In two of our test worlds, Virtual Valley and Neighborhood, map goodness and object class number scores correlated significantly with the subjects' self-reported sense of orientation within the virtual world. The relative object positioning ratio also matched the difference in reported orientation between Virtual Valley and Neighborhood worlds. These two results suggest that sketch maps do indeed accurately represent the topological aspects of subjects cognitive maps.

    The "significant object" ratio appears useful for comparing across worlds, while the map goodness and object class scores are useful for comparing subjects within worlds. The difference between the "significant" and "all" object placement ratios may also be used to identify worlds that have well defined landmarks.

    However, the low correlation with the Cloudlands results may indicate that sketch mapping is more useful for relatively dense worlds, or that more complicated forms of sketch map analysis is needed for sparse worlds.

    ACKNOWLEDGMENTS

    The authors would like to thanK the reviewers for their useful comments, and Hunter Hoffman and William Winn for help with analysis of the experimental results.

    REFERENCES

    1.Alfano, P.L. and Michel, G.F. Restricting the Field of View: Perceptual and Performance Effects. Perceptual and Motor Skills, 1990, 70,pp3545.

    2. Appleyard, D. Styles and Methods of Structuring a City. Environment and Behavior, 1970, vol 2, pp 100-118.

    3. Blades. M. The Reliability of Data Collected From Sketch Maps. Journal of Environmental Psychology, 1990, vol 10, pp 327-339.

    4. Briggs, R. Urban Cognitive Distance. In Image and Environment, (Downs, R.M. and Stea, D. Eds.), Aldine Publishing Co., Chicago, 1973,pp361-388.

    5. Darken, R.P. and Silbert, J.L. A Toolset for Navigation in Virtual Environments. In Proceedings of the ACM Symposium on User Interface Software and Technology, UIST'93, 1993, pp l57-166.

    6. Downs, R.M. and Stea, D. Cognitive Maps and Spatial Behavior. Process and Products. In Image and Environment, (Downs, R.M. and Stea, D. Eds.), Aldine Publishing Co., Chicago, 1973, pp 8-26.

    7. Golledge, R.G. Methods and Methodological Issues in Environmental Cognition Research. In Environmental Knowing, (Golledge, R.G. and Moore, G.T. Eds.), Dowden, Hutchinson and Ross, Inc., Pennsylvania, 1976, pp 300-313.

    8. Griffin, D.R. Topographical Orientation. In Image and Environment, (Downs, R.M. and Stea, D. Eds.), Aldine Publishing Co., Chicago, 1973, pp 296-299.

    9. Henry, D. Spatial Perception in Virtual Environments: Evaluating an Architectural Application. M.S.E. Thesis, College of Engineering, University of Washington, 1992.

    10. Kuipers, B. The Cognitive Map: Could it have been any other way. In Spatial Orientation: Theory, Research and Application, (Pick, H.L. and Acredolo, L.P. Eds.), Plenium Press, New York, 1983, pp 345-360.

    11. Ladd, F.C. Black Youths View their Environment: Neighborhood Maps. Environmental Behavior, 1970, vol 2, pp 64-79.

    12. Lynch, K. The Image of the City, MIT Press, Cambridge, Massachusetts, 1960.

    13. Moore, G.T. Theory of Research and the Development of Environmental Knowing. In Environmental Knowing, (Golledge, R.G. and Moore, G.T. Eds.), Dowden, Hutchinson and Ross, Inc., Pennsylvania, 1976, pp l38-164.

    14. Neisser, U. Cognition and Reality, WH Freeman, San Francisco, 1976.

    15. Newcombe, N. Methods for the Study of Spatial Cognition. In The Development of Spatial Cognition, (Cohen, R. Ed.), Hillsdale, New Jersey, Lawrence Erlbaum Associates, 1985, pp 277-300.

    16. Stea, D. and Blaut, J.M. Some Preliminary Observations on Spatial Learning in School Children. In Image and Environment, (Downs, R.M. and Stea, D. Eds.), Aldine Publishing Co., Chicago, 1973, pp 226-234.

    17. Walsh, D.A., Krauss, IK., and Reginer,V.A. Spatial Ability, Environmental Knowledge, and Environmental Use: The Elderly. In Spatial Representation and Behavior Across the Life Span: Theory and Application, (Liben, L.S., Patterson, A.H. and Newcombe, N. Eds.), Academic Press, New York, 1981, pp 321-357.

    18. Weghorst, S. and Billinghurst, M. Spatial Perception of Immersive Virtual Environments, HIT Lab Technical Report, University of Washington, 1993.

    APPENDIX: SUBJECT SURVEY

    The 24 survey questions given to subjects are listed below. For each of the questions subjects were asked to rank their responses on a scale from one to ten. The anchors for these scales are shown under the each of the questions. Responses were collected automatically using a Hypercard stack on a Macintosh computer and participants were also given the opportunity to add their own comments at the end of the survey.

    Questions

    1. Sense of being there:

    None - Total

    2. Ease of interaction:

    Impossible - Effortless

    3. Comfort of the display hardware:

    Unbearable - Comfortable

    4. Enjoyment:

    Boring - Very enjoyable

    5. How easy was it to navigate?

    Very difficult - Very easy

    6. Sense of orientation relative to the laboratory:

    No sense of direction - Completely orientated

    7. Sense of orientation in the virtual world:

    No sense of direction - totally orientated

    8. Feeling of being lost:

    All the time - Never

    9. Sense of dizziness:

    Never - All the time

    10. Image brightness:

    Way too dim - Way too bright

    11. Color quality:

    Very poor - Very good

    12. Ease of use of the glove:

    Very difficult - Very easy

    13. Feeling of inclusion in the world:

    Totally removed - Actually there

    14. Overall physical comfort:

    Very uncomfortable - Very comfortable

    15. Understanding of where everything was in the world

    Total confusion - Total understanding

    16. Invites exploration:

    Not at all - Very much so

    17. Invites introspection:

    Not at all - Very much so

    18. Ease of movement around the world:

    Very difficult - Very easy

    19. Ease of getting where you wanted to go:

    Very easy - Impossible

    20. How engaging was it?

    Not at all - Totally

    21. Image clarity:

    Extremely fuzzy - Extremely sharp

    22. How comfortable are you with using computers?

    Totally uncomfortable - Totally comfortable

    23. Your experience in Virtual Reality:

    First time - Very Experienced

    24. Sense of presence within the Virtual World:

    Very low - Very High