From: "PSOTKA, JOSEPH" <psotka@alexandria-emh2.army.mil>
Subject: PSYCH: Psychology and VR
Date: 7 Jul 92 22:22:00 EDT


[Co-mod (Mark): This article was a bit mangled.  I've taken the
liberty of removing a lot of word-processor control characters from it.]

Although I have now seen the quote many times: "Psychology is the
physics of VR", or something like that, I too have noticed that this
discussion group is dominated by talk of hardware, and not peopleware.
As a cognitive psychologist beginning to enjoy the hardware, even in
its generally primitive state, these discussions are interesting and
useful.  But there are so many psychological issues too: the
interaction between memory and perception in this environment is
particularly exciting!

The Event:

Recently, while I was chasing fish in this environment (and just
beginning to catch them) someone in the *real* world pushed me gently
out of the way of some (to me) invisible obstacle in the *real* world.
This set off a state of confusion that was particularly difficult for
me to understand and explain.  I am not sure I have the best
explanation yet.

My Confusions:

My first confusion was apparently that I thought that if I were moved
in the *real* world I would NOT be moved in the VR environment.  This
was wrong and created a conflict of expectations when I realized I WAS
being moved.  It also created another conflict when I realized that I
could not move to avoid being attacked by a shark headed towards me
and that I would miss it with my net.  Since I WAS in fact moved in
the VR environment, there was some confusion about how to reconcile
this with my environment: a strange *action at a distance* by
invisible forces.  Somehow I recognized the violation of conservation
of causality in the VR environment and was confused by it.  But I was
also frustrated by not being able to share this state with my *mover*
in the real world who could not see my predicament even though he was
standing right beside me!  This left me confused about what I could do
to communicate with him.
	My second confusion was that I thought by moving me, I would
lose track of where I was in the *real* world.  In fact, I had already
lost track of where I was, so this added motion was not going to make
any diference.  But somehow I was under the misapprehension that I had
not moved in the real world and when I took my HMD off, I would find
myself in the same place where I had been.  Obviously this was a
momentary confusion, and something *illogical*; but I point it out not
to show how stupid I am, but to present something of an insight into
the default workings of our *presence* analyser when it is preoccupied
with dual reality tasks.
	My third confusion arose because the hands around my waist
moving me also pulled me back into that reality.  This was the real
source of confusion, because literally i could see myself swapping in
all the memories of my *real* surroundings into the *presence* store,
and quite visibly swapping out the vision of the fish and water in the
VR.  I began *seeing* in my mind's eye the computer monitors and
people standing around me, and of course that meant *not seeing* the
fish and water.  But at the same time I realized how useless my vision
of all those real surroundings were.  From their voices I could
roughly tell where the people were but some were silent and so it was
not possible to locate them accurately in space.  Besides, my point of
view (POV) or frame of reference (FOR) was still the old one when I
first put the HMD on, and I realized I had no idea where I was
relative to that FOR.  I had immersed myself almost immediately in the
VR and lost all track of my motion.
	So there was nothing more to do than complain loudly that I
didn't need any help and get on with chasing the fish.  But of course,
I now had to redefine my POV in the VR.  That meant swapping out again
all the old representations of the *real* world and scanning the VR
again to find where I was and where the fish were.

Some Speculations

The momentary *distraction* and confusion must have lasted several
seconds.  It seems in retrospect that I could not hold these two
realities intact separately.  There seems to be an all or none store
of our immediate surroundings.  In order to retrieve the memories of
one reality after dealing with the other there appeared to be a
realtime delay and a difficult period of swapping memories from one
memory into another. This was a relatively slow process, bandwidth
limited and possibly serial, driven by salient cues in each
environment.  For instance, the hands on my hips led me to seek to
identify the owner, who conveniently spoke.  That led to memories of
his position, my earlier FOR and the salient items -- e.g. monitor,
doorway, cables , wall --- that came flooding in with the FOR.
	Could it be, that with practice we could become more adept at
holding these two realities intact, synchronically?  I seem to doubt
it.  If anything, my experiences in VR are making it easier and easier
to enter VR completely immersed, and easier to forget entirely my real
world surroundings.  On the other hand, it is easier to leave and
return completely too.  It seems that inexperience with VR results in
greater intermingling of the two realities, holding on to one while
experiencing the other.  A little practice makes it easier to believe
that dual realities or even multiple realities are possible: a
liberation of the imagination; even an UNwilling suspension of
disbelief.

The Memory of Presence

The intriguing possibility this switching among realities suggests is
that there is a special memory mechanism or store that deals
specifically with our surrounding *presence*.  It is filled up as we
scan and touch and listen to our surroundings.  When we enter another
VR it gets overlaid with the sensory experiences of that VR.  If the
VR is brand new, the overlay of the *Presence Store* is perhaps
piecemeal, overlaying parts of the earlier reality as it is
discovered.  If the new VR is one that has been experienced before, it
appears to be pulled in from long term memory in a default mode.
	There is an intriguing relationship and similarity between
this *Presence Store* and what psychologists have traditionally called
short term memory.  Could the two be one?  It seems to me that
psychologists will have a great deal of fun over the next few years
teasing the similarities or differences among these possibilities
apart.

An Experiment

This suggests an interesting experiment.  As one enters a new reality,
looking at it through the aperture of a (typically) 90 by 60 degree
FOV, does the new reality literally push out the old?  Is there some
sort of push down stack of realities?  Is the old simply put into long
term memory, or some sort of intermediate working memory store where
it can be recalled easily?
	Imagine creating a real environment with fixed list of items
in the surround; e.g. a chair to the right, a picture ahead; a table
to the left.  Now imagine a VR with similar items in sligthly
different places.  A different chair, picture, table.  Now imagine
that the reality traveller is exposed to the VR only one hemifield at
a time.  S/He sees only the VR on the right and black space on the
left.  Does the black space push out the memory of the old reality as
much as the half-room with its items?  Would reaction time differences
to pointing at items in the old reality show up some differences in
the way one reality replaces another?  Are there interactions between
the two realities? -- one chair and the other; one table and the
other?
	 Is the split reality as effective an immersion environment as
a whole reality?
	If the VR were shown with smaller FOVs, could one detect the
localized effects of the scan pattern?  e.g.  if the scan were
systematic and complete, would it be more effective at pushing out the
old reality than a scan with small FOV ( say 20 by 20 degrees) that
waved erratically around the scene leaving splotches unscanned.  Would
the parts of the scene not scanned act like a hand on the shoulder,
pulling the traveller back into the other reality?
	Is this presence memory something that can be localized in the
parietal or inferotemporal cortex?  It is suggestive that Penfield's
stimulation of the inferotemporal cortex brought about *presence*
feelings of hearing voices and seeing objects *as if they were there!*

Let's get to Work

Psychologists should have a field day with some of these questions.
Answers to some of them and many more begin to flesh out spatial
representation, imagination, perception, and memory.  They bring to
the study of sensation and perception an augmentation of importance
and relevance to daily life that will be an attraction and motivation
for some and a distractor and distasteful to other researchers.
Nevertheless good research will be possible on these topics and
enlightenment will be found that will help us to design more
meaningful and entertaining realities, even realities that could
educate us about physics, culture, and even psychology.

Joe Psotka  (:8-<>}