| Publications Page | HITL Home |
Any technology which has the audacity to call itself a variety of reality must also propose a paradigm shift. In essence, a paradigm shift expands the potential of an entire discipline. For me, VR has expanded every aspect of Computer Science, and is providing a base for a very satisfying philosophy as well.
And just what is the paradigm shift? Computers are not only symbol processors, they are reality generators. Until recently, computers have generated only one dimensional symbolic strings. Text and numbers. Text is a code which, when read, generates images of reality in our minds. During the 80s, we enhanced the expressability of computation by adding space and time dimensions to the realities being generated. Two dimensional windows, 2D animation, solid modeling, simulation. Now, in the 1990s, computer systems can generate virtual environments, entire multisensory worlds which include us as interactive participants. Digital information can seem as-if-real, changing our notions of computation, symbolism, meaning, metaphysics, self, and culture. Virtual realities are more than real.
The potential for VR to contribute to societal infrastructures such as manufacturing, marketing, telecommunications, science, entertainment, art, education, medicine, and media, suggests an economic impact that rivals the Gross National Product. We live in two superimposed worlds, the one of mass and the one of information. The huge accumulation of difficult to access words on paper indicates that the world of mass is not particularly well-suited for dealing with information. As our culture matures into an information society, we are now discovering the virtual world, an ideal place for interacting with information.
What follows is a wide ranging discussion of interesting growth areas for VR. I'll define VR, point to some active areas of research, tell you about virtual world tools, outline some things we have learned from working in the field, and discuss some risks and philosophies engendered by VR techniques.
Behavior Transducers hardware interface devices Inclusive Computation software interaction techniques Intentional Psychology biological constraints and plasticity
Behavior transducers map natural behavior onto digital streams. Natural behavior is what two year olds do: point, grab, issue single word commands, look around, toddle around. Behavior transducing interface devices include body trackers, voice recognizers, spatial sensors, kinesthetic feedback devices, and subjective audio and video displays. Transducers work in both directions, physical behavior to digital information (the virtual body) and virtual display to subjective experience (the physiological model).
Inclusive software provides tools for construction of, management of, and interaction with digital environments which surround a participant/user. The central design issue for VR is getting behavior transducers and virtual environments to feel good to a participant. The intentional psychology of VR will require a deep knowledge of how we work, our physiology, our sensations, our cognition. We must refocus the effort of interface from the needs of symbol processors to the needs of people.
picture --> place observe --> experience use --> participate interface --> inhabit
When we extend our field of view onto a computational environment beyond about 60 degrees, a remarkable phenomenon occurs. We shift from a feeling of viewing a picture to a feeling of being in a place. This shift is accompanied by an emotional response. It seems as though the unification of our symbolic processes with our visual processes creates a feeling of wholeness, of empowerment. We shift from external users (exercising rights) to internal participants (exercising responsibilities), from being observers to having experiences, from interfacing with a display to inhabiting an environment.
My colleague Meredith Bricken and I have collected videotaped behavior and exit interviews from over 500 people that we have guided through initial VR experiences. We have seen overwhelmingly positive responses, eagerness to return to "that place", willing suspension of disbelief.
information database, sci.virtual-worlds simulation laboratory virtual environment operating shell laser microscanner display techniques design and construction of worlds 3D audio display instrument display prototypes multiple participant worlds educational experiences and environments virtual prostheses
The information database is a project for NASA to follow the development of VR and to serve as a clearinghouse for references and research in the field. Sci.virtual-worlds is a moderated USENET newsgroup for the discussion of VR issues.
The simulation laboratory provides a research environment for prototyping VR hardware and for testing and evaluating effects on human sensory, perceptual and psychomotor behavior.
The Virtual Environment Operating Shell is a software suite currently written in C that wraps around the UNIX operating system. VEOS provides resource and communication management for coordination of the modules which make a VR system:
i/o hardware, behavior transducing input and display devices world construction kits, CAD packages dynamic simulation kits, for interaction and animation virtual world tools computational and display processors
The laser microscanner is a hardware research project to design a high performance, low cost virtual display. Rather than creating an aerial image using cathode-ray tubes or matrix element devices, the laser microscanner scans a color image directly onto the retina. We don't think in terms of addressing pixels, we think in terms of addressing rods and cones directly. The head-mounted unit will integrate 3D visual and audio display, voice recognition, and head and eye tracking.
We build worlds for presentation, evaluation, and experimentation. Our interest is the design of comfortable, functional worlds.
For Boeing, we are exploring 3D audio display techniques, and building prototypes for design and display of complex instrument panels and machines, in essence simulating the design of aircraft cockpits.
We are working on the implementation of multiple participant worlds for an application to telecommunications. You can think of VR as a very sophisticated replacement for the telephone.
Education and industrial training are natural applications of VR techniques. We are designing virtual environments conducive to learning, we're studying the transfer of skills between virtual and actual tasks, and we're exploring the implications of VR for educational theory and practice.
And we have great interest in the application of VR to prostheses for the handicapped, for providing virtual bodies which extend individual capacities, for providing alternative control devices for interaction in virtual worlds.
audio modeling teleoperation, telepresence image integration, HDTV interactive drama military simulation
3D audio hardware is commercially available, we should expect to hear of inclusive sound systems in the stores soon. Audio theorists are interested in specification languages for 3D music, in audio lenses and icons (earcons), and in modeling ambience, the analog of ray-tracing for sound.
Telepresence, the development of remotely controlled robots, requires the same interface techniques as VR. The primary difference between these disciplines is that teleoperation looks at interaction with real (usually inaccessable) images, VR looks at virtual images. Both want inclusive, interactive environments. The possibility of inhabiting real worlds shook me out of a self-imposed computer graphics narrowness. We can apply VR interaction and hardware techniques to explore anywhere we can place a probe. We can inhabit a remote undersea vehicle, processing digitized images into worlds that mix the actual with the virtual. We can swallow a miniaturized transmitter and explore our own stomach. We can build artificial bees with fiber optic visual links and micromotors for dancing and for rubbing antennae. We can then put our virtual bee-selves into the physical hive and interact with real bees in their home environment. I can hardly wait.
The multimedia community is very interested in digital images. It seems only natural that we should port these flatlander tools into VR. We could tile polygons with TV. More importantly, automated conversion of images to 3D objects (the image recognition problem) would permit a seamless integration of video-real with graphic-virtual.
Hypertext has raised the question of interactive fiction. The theatrical community is working to install plot and character into virtual worlds, creating interactive drama. What do a good story and a good experience have in common? Can we construct participatory plots, guided experiences, autonomous characters?
Actually, VR grew up in the military. The first substantive application of VR was to help Air Force pilots improve their ability to aim missiles. The most refined and widely distributed VR environment today is SIMNET, a large scale, simulated tank combat system. Recently, I saw a paper on training close combat fighters in VR. Sort of reminds me of the video arcade.
the wand the virtual body virtual home, virtual community concurrent inconsistent worlds autonomous entities concrete mathematics, experiential programming
The Wand is an evolution of the Mouse. It is a simple physical device with a wide diversity of uses, ideal characteristics for a tool. Physically, the wand is a spatial position and orientation sensor on a handheld stick. In software, the Wand emanates a ray which can be used for pointing at virtual objects. Coupled with voice commands, the Wand can be used to identify objects, to attach to and move objects, to bring things closer or place them at a distance, to indicate a direction for flying, to identify a location to teleport to, to measure distance, as a pen for drawing, as a knife, as a switch, as a spotlight. Lots of functionality from a little hardware.
People achieve presence in VR by inhabiting a virtual body. The virtual body is a software toolkit for associating an arbitrary suite of behavior transducers (such as wands, voice command systems, headtracking, etc.) to a display of self in a virtual world. What we do physically is sensed and converted to virtual behavior. Don't think that the virtual body is necessarily in the shape of our physical body; any object in VR can be inhabited. If you are controlling a physical robot, you may prefer your virtual body to be the shape of that robot. If you are navigating a data terrain, you may prefer to have a virtual body shaped like a jeep or an airplane. The virtual body can filter and map physical behavior onto superhuman capacities. One of the first things we did to figure out how a virtual body might be used was to search the old comic books for super powers.
The virtual home is an environment designed for personalized comfort, for work and for play. My virtual home will have a cozy chair, a fireplace, some cats, and a cabinet full of virtual tools and toys, essentially what I now have at (physical) home. Physical reality is a great starting model for virtual reality. Take what we like and delete what we don't.
Virtual homes will be customized, personalized environments. The virtual home extends to a virtual community. People we work with are not organized by some cryptic email address that is basically a program to tell the network where to find them. They are organized in close proximity in space. In a virtual community, friends have virtual homes that are visible from our own virtual home. They are our neighbors. We visit them by pointing to their home and saying "jack me there". Less frequent acquaintances may be down the road or over the hill. The idea is to organize virtual space to accommodate to human culture.
One profound capability in VR is to maintain inconsistent views for different participants, to intermix personal realities. In physical reality, mass has a way of being unarguable. We quickly default to assuming a consistent, objective reality that is communal to everyone. Consistency is an assumption and is widely overgeneralized. Each person in physical reality, for example, has a viewpoint, each viewpoint is necessarily in a different physical place, each perspective provides different information about the inclusive environment. Every experience is unique. We agree to suppress our differences for massive objects, but the line is always fuzzy. We certainly tolerate differences within the domain of conversation. How we talk is an excellent example of concurrent inconsistent worlds.
In VR, communality can be negotiated rather than assumed. In VR, the color of my shirt can appear to be green to me, but blue to you. So long as we do not talk about or interact with the color of the shirt, how it is rendered to each of us is irrelevant. Carry this a bit further: I can be sitting in my virtual home next to an empty chair. You jack a duplicate of your virtual body into that chair. From my perspective, you are visiting me. Now, from your perspective, you are still sitting in your virtual home, in your customized environment. You have an empty chair, and I jack a duplicate of my virtual body into it. We are now sitting in two totally different environments while sharing a mutual conversation. For me, you are in my home, for you, I am in your home. So long as the inconsistencies in our environments are not items of contention or confusion, the differences will not interfer with communication. When they do interfer, the explicit differences become subject to negotiated resolution.
But the pluralism of VR is much deeper. It is possible to maintain inconsistencies directly, without resolution, using a mathematical technique called the imaginary boolean value. We could choose to represent the color of my shirt as ambiguous, as context dependent. Both green and blue. We can then discuss the color of the shirt as being inconsistent, as information about which we simply do not see eye to eye. I bring up these ideas from an esoteric branch of representation theory to illustrate a fundamental point. VR is not bounded by the assumptions of physical reality. We can have whatever we can formally specify.
The HITL architecture specifies that every object in VR, including space itself, have processing and memory resources. Entities are objects with the capabilities of operating systems. Every entity is a system, every entity is a variant of the same system. This means that we can use the same editing, debugging, and interaction tools for modifying each entity. Entities are running a sense-process-act loop; in artificial intelligence terms, each entity is an agent, an actor. This means that VR is inhabited with artificial life. Every entity is capable of independent action, in response to environmental changes, in response to internal memory or process changes, or in response to changes in the rules, the disposition, specifying that entity's internal processes. Each entity is an expert system using pattern-matching on its input to trigger disposition rules and metarules which generate outputs to the context. The environment itself is just another entity, one that includes other entities within it. All cyberspace is Toontown.
We have been able to demonstrate that mathematics itself (in particular logic, integers, and sets) can be expressed concretely, using 3D arrangements of physical things, such as blocks on a table, doors open or shut, rockwalls that respond to gravity, the things of everyday life. String-based symbolic representations of mathematical concepts are typographically convenient, but tokens are not at all essential to mathematical expression. VR makes it convenient to express abstract ideas using spatial configurations of familiar objects. One benefit of this approach is that we can build visual programs, set them on a virtual table, and watch them work. We can experience programs as other entities rather than as dumps of text. Bugs would manifest as structural anomalies, as visual irregularities. Architectural design has a sensual, experiential semantics. It is but a quirk of typography that we have ignored the experiential semantics of computational languages. More fundamentally, experiential computing unites our spatial and our symbolic cognitive skills, permitting mathematical visualization, analytic gestalt, whole brain processing.
Constructivism "Human knowledge is essentially active." Piaget Natural Semantics non-symbolic, preoperational interaction Programmable Participation conducive and responsive environments Cognitive Presence modifiable self-concepts, learning by becoming Social Reality unique concurrent worlds
Educational psychologists have long known that people actively construct their experience of reality. In VR, students will construct their knowledge, then dwell within it, exploring their understanding.
Natural semantics means that the computational environment hides symbolism in favor of displaying information in an innately recognizable form. The two-year-old criterion: if a kid recognizes it, its natural. The three Rs, all symbolic, will become the three ACTs: enact, interact, and abstract.
VR provides the potential for completely customized, individualized learning. Educational environments will uniquely respond to the participant-learner, in terms of both needs and preferences. A student model will not be necessary, instead the teacher and student will modify the environment in support of student behavior.
We also have a tool for affective education, for sharing perspectives, for mapping perspectives into broader contexts, for changing self-image, for remapping capabilities.
Education is inherently social. Explicitly shared worlds and multiple concurrent agreements provide the opportunity for groupwork, social consensus, and the construction of functional, multiparticipant environments.
In general, everything we do to educate with words and pictures can be provided as virtual experience.
Psychology is the Physics of VR. Our body is our interface. Knowledge is in experience. Data is in the environment. Scale and time are explorable dimensions. One experience is worth a trillion bits. Realism is not necessary.
A major theme of VR research is that Psychology, in the broad sense of behavior, perception, cognition and intention, provides the rules and the constraints of virtual worlds. Psychology is the Physics of VR.
This may come as a shock, it is one of those truths that is obvious after it is said, but elusive before it is stated explicitly. Our body is our interface. Interface is not something that is out there, in some machine. Interface is a boundary which both connects and separates, interface takes place at the surface of our skin. From the perspective of VR, interface is physiology, interaction is natural behavior. We simply want to use the power of computers to make computation invisible.
Knowledge is in experience, it is not in some abstract, symbolic representation. Data is in the environment, it is not stored away in some memory array. These observations serve to remind us that we are not the computer. To understand computation, we should participate within it, rather than writing programs to dominate it. Humans have a great skill for projecting outward, for becoming the tool we are handling. We need reminding that we are creatures who dwell inside an environment.
VR is inherently multidimensional. As well as freedom of translation and rotation, in VR we can travel in scale and in time. Think of scale as simply another direction; when we traverse scale, size instead of location changes. We can also travel through time using any of the techniques of film editing, including slow-motion, fast forward, and temporal discontinuity.
There is a tremendous compression ratio between digital information and human experience. Very approximately, it takes a hundred million polygons to simulate what we see in one scene. Add duration, multisensory channels, and interaction, and you get a lot of digital information being transacted with each moment of consciousness. Computation will not come close to this bandwidth for a long time. Fortunately, virtual world experience does not require the information density of physical reality.
Because our minds provide such tremendous flexibility in interpreting what is outside of us, realism in VR is simply not necessary. Our cognitive plasticity permits even simple cartoon worlds of 500 polygons to be experientially satisfying. We must design worlds that respect our physiological needs. For example, we conceptualize perspective in physical space as having six degrees of freedom, three in translation and three in rotation. But our bodies have roughly four and one-twelth degrees of freedom. We move easily in all directions on a plane, forward and to the side, but not up, off the surface we stand on. We rotate freely around the vertical by turning, but our natural rotation forward, around our waist by leaning, is at best 270 degrees (3/4 of a full 360 degree rotation). And our ability to bend side to side is only about 120 degrees, one-third of a full rotation. This adds up to a little more than four degrees of physiological freedom. Input devices which permit complete freedom of translation and rotation usually get people lost in space. The dimensionality of our abstract perspective does not match that of our physical construction. We must also differentiate that which is innate from that which is learned. Pilots, for example, have learned to fly in all six degrees of freedom. Realism is both physiological necessity and cognitive interpretation. In VR, world design that conforms to physiological necessity frees our minds to furnish the rest of our reality.
public domain VR software massive database access fabric of space negotiable group space conversational programming artificial life crossvalidation of realities
HITL is electing to distribute its software in the public domain. We hope to create a context for the growth of an industry and for the understanding of alternative realities. We hope to encourage the evolution of a shared software and hardware environment which will permit researchers to share progress and results. The commercial marketplace can then improve on public work, selling value-added features like customer support, prebuilt worlds, faster hardware, better algorithms, realer time.
VR requires a new approach to database management. We want to access massive databases such as Landsat as a function of our perspective, our location in the database. We expect to see interactive databases which we can explore through movement. Already waiting is the entire Earth to one meter resolution, the location of every aircraft and ship, large hunks of the Moon, the human body down to the resolution of a cell, the flow of the economy, the network of computation. We have digital worlds to explore.
I have mentioned that space is an entity. Many interactions between entities can be expressed as internal processes of the spaces which include them. Gravity is a primary example; we can implement simplistic local gravity by decrementing the Z component of the velocity vector of each entity in a space at each time tick. Rules that apply uniformly to every entity in a space instead can be ascribed once to the space itself. The inclusive space enacts local gravity by owning the locations of the entities it includes. We want to be able to place fields in space, to have space maintain its local version of continuity, gradient, and metric, to build space-filling logics which branch as a function of location.
One advantage of customized environments is that we will have to be explicit about what is shared. VR suggests an approach to cooperative work in a computational environment: rather than assume communality and specify differences, assume complete difference and specify what is common. It may turn out to be fun to build communal, consensual contexts, to negotiate the group space.
One consequence of autonomous entities is that they can respond to our communications. With voice recognition, we will be able to speak to virtual entities as a means of programming their structure and behavior. "I want the green cube I'm looking at to double in size." The cube has a sensor for voice. Its rulebased disposition matches the vocal input to its own identity and to its size changing function. If you have permission, it changes itself to your specification.
Another consequence of autonomous entities is that they may have their own agenda. The coupling of the behaviors of several entities could determine events. Rulebases that support emergent behavior are tremendously difficult to construct. We hope that the programmable environment of VR will provide autonomous entities with a context for the growth of interesting virtual life.
Fundamentally, VR forms a new reality, at least to the extent that we are willing to relax our minds. We will need to calibrate the effects of transfer across worlds and across realities. VR is the first empirical tool of metaphysics, it permits us to compare realities, to ask which alternative reality is preferable for which tasks.
descriptive confusion lack of experience cognitive remodeling fluid self sensory overload, sensory ecstasy power and control cultural adaptability
VR is seeking definition, it could be anything from email to a fully surrounding, multi-sensory environment. We are struggling with appropriate comparisons. VR is not a drug and is not physically addictive. Drugs change our perspective from inside the body, VR changes our external environment. VR may well be psychologically addictive (that is, entertaining), just like all good media experiences can be. And there is that constant tension between physical responsibility and cognitive exploration. Is VR escapist? Escapism means seeking diversion from physical reality. VR cannot escape being escapism, VR is perfect escapism. Is VR theater, or interactive drama, or is it more than art? Is it scientific visualization, or physical simulation, or is it more than science? Is it financial modeling, or the perfect sales tool, or is it more than economics? It's a good idea to spend some time figuring out what VR is.
To me, the greatest problem is that we have virtually no experience in VR. There are perhaps around ten thousand VR non-virgins. But I estimate that there are no more than fifty people who have spent twenty hours in VR. All of this excitement is purely conceptually, we have very little experience with what we are talking about. The first item on the VR agenda must be to construct and distribute hundreds of systems, so that many people can contribute to our understanding. We should know at least something about the cognitive effects of VR before it is a consumer item with the distribution of Nintendo. When a representative of MCC asked the lab the best way to invest two million research dollars in VR, the answer was clear: give away forty $50,000 systems.
The most complex, and potentially dangerous, risk is what we are calling cognitive remodeling. Those who spend a lot of time in VR bring back to physical reality some strange habits, like navigating across a room by pointing, like bumping into walls cause they aren't just images, like dreaming in polygons. VR effects dreaming strongly, it seems to provide tools for control of the dreamlife from within the dream. VR changes mental models. Now, it is not dangerous that this is happening, cause all intense work produces similar effects. Anyone who has programmed all night will know that the programming slips into dreams. The problem is not that these things happen, it is that we don't have the faintest clue what is going on. We do not know the borders between virtual and actual. We have not yet had the opportunity to evaluate current theories of reality crossing.
And how will we react when we are able to redefine our bodies, swap our perspectives, mix our senses. We will have the ability to map arbitrarily across sensory input, self-image, and behavioral output. What will a fluid self be like? We will need to understand the cognitive and behavioral effects of transportable perspectives, of programmable bodies, of synesthetic sensations, of exchangable body parts, of inhabiting arbitrary objects, of masslessness, of negotiable communality, of complete empowerment.
Are there limits to the degree of warpage our senses can tolerate? This is, of course, an empirical question. What are the functional constraints of sensory modification for enhanced productivity, for enhanced enjoyment? Are there sensory pathways to insanity or to ecstasy? Just which side of the monitor do you stand on?
We have been discussing a domain which emphasizes personal freedom. VR could be used for horrible purposes, but that negative assumes that we are strapped to a chair. So long as each individual has the freedom to reach up and turn off the experience, VR itself is quite benign. But how will authority respond to this frontier? VR is interactive, but will I have the right to remove the virtual arches in my prebuilt reality given away with each hamburger? Are advertisements from the creator necessarily non-interactive? Where are the edges of property and ownership in a world which is digital? Will there be commodities? What are the rights of autonomous computational entities? Will there be stability? Will there be a Virtual Environmental Protection Agency? I don't know, but I certainly look forward to negotiating the communal rules of personal responsibility in cyberspace.
The biggest issue is how our culture will respond to this new reality. We have amassed hundreds of years of favoritism for the objective, the scientific. Our values, ethics, and aesthetics are predisposed toward Objectivism. Is VR a better place for transacting information? How will physical reality react to competition? What will socialization without material consequence be like? What kind of intimacy will arise from explicitly penetrating world views? What kind of cultures will arise when the VR network is standardized? Are we like Columbus, discovering a completely new land in an unexpected place? Is living in VR necessarily pathological? These are indeed exciting times.
situated semantics pervasion immaterial realism constructivism boundary mathematics more than reality
Situated activity is a growing school of thought in AI. The idea is simple: what we do depends on our environment as well as our internal state. We react and respond, constantly bringing external context into our interpretation of the moment. Currently, symbolic logic is split in half, between syntax (representation) and semantics (meaning). Syntax is strictly formal, it has no basis in experience. Semantics attempts to connect syntactic symbols to the reality of the world by mapping representation onto meaning. The problem is that it does so without regard to context external to the formal symbols. Since environments necessarily introduce external unknowns, standard semantics is just too literal. Situated activity is an attempt to build a theory of context.
VR, in comparison, is totally situated. By defining natural behavior as the rules of interaction, by displaying recognizable spatial structures as output, by providing context in toto, and by including the participant, VR redefines the relation between syntax and semantics. Semantics, what we consider to be anchored to reality, is displayed directly as (virtual) reality. Syntax, the symbols that guide computational activity, is hidden in the background, out of sight.
Environments include their participants, they pervade their contents. Pervasion is a non-dualistic concept, more familiar to Buddhism than to Christianity. A pervasive space is one which is diffused throughout every part of itself, including those parts occupied by other spaces and objects. Objects themselves are those boundaries of spaces that we can sense. When we look at the container of an environment, from the outside, we see that it surrounds a portion of space. The important point to understand is that environments focus our attention on a particular portion of space, they do not separate space into two opposite parts. The outside space still pervades the inside space. For VR, the physical pervades the virtual. When we enter a virtual world, we always bring our physical body. VR is not a separate reality, in a dualistic sense, it is a pervaded reality. The shift from duality to pervasion is from networks to maps, from separation to unity, from confrontation to cooperation, from male to female, from one to zero.
The Copernican revolution introduced a physics that differed fundamentally from appearance. VR introduces a metaphysics that differs fundamentally from the material. At the foundation of Objectivism is an attempt to be realistic about the material world. VR calls for immaterial realism, for being realistic about information. The currency of VR is organization, not possession, not accumulation, not territory. All laws are transmutable, we can satisfy fantasy rather than fact. It is science itself that is redefined. In VR, we can choose to be reductionalist, but at the bottom of it all, there is not Mass or Nature, there is the Void. VR is representational, but not a priori rational, empirical, or verifiable. VR is illogical positivism: if you can specify it, it is meaningful. All empirical hypotheses are true.
Another fundamental philosophical position engendered by VR is that of constructivism, that our minds and our bodies coparticipate in defining reality. Objectivism places an overemphasis on the input of the physical body. Solipsism overemphasizes the mind. Constructivism recognizes that reality is like light, it is both particle and wave, both objective and cognitive, both observation and participation. In VR, we cannot escape the realization that we are the architects of our environment.
We are applying boundary mathematics to VR in three different ways: as a foundational mathematics, as a technique for logical deduction and maintenance of inconsistency, and as a spatial embodiment of abstract concepts.
Boundary mathematics is a calculus of inclusion. The essence of VR is inclusion, the relationship between an environment and a participant. The primitives of boundary mathematics are also participant and environment. Let ( ) represent an environment, and let i represent a participant. A variation of Spencer Brown's Laws of Form provides the axiomatic basis:
Observe: i ( ) = ( ) Participate: ( i ) =
The left-hand-side of each equation is descriptive (objective), explicitly mentioning the participant. The right-hand-side is experiential (participatory), implicitly using the participant's perspective. We read the left-hand-side from our traditional externalized, objective perspective. The right-hand-side refers to our experience, from the subjective perspective. When we observe an empty environment, we perceive its boundary. When we are included in an otherwise empty environment, we perceive emptiness. That's all there is at the foundations of experience.
The most important thing to realize about VR is that it is more than reality, more than a simulation of reality. You add physical realism to a virtual world by adding constraints that reduce the possibilities in that world. Native VR lets you walk through walls, we add collision detection to disallow this power. Native VR has no gravity, we add gravitational equations to simulate a gravitational reality. Reality simulation is a subset of potential VR experiences. The least elaborated virtual world is the Void.
We describe innovations in terms of what they replace. Only after decades do we come to understand the pervasive impact of new technologies on our culture. The automobile was first the horseless carriage. It replaced the carriage, looked like a carriage, and moved at the speed of a horse. Decades later, the automobile has transformed our landscapes, the pace of our travels, and our concepts of time and space. The television replaced the radio. Television programs were first radio programs with pictures. Decades later, the television has transformed our evenings, the pace of our senses, and our concepts of news and entertainment.
The computer is first a symbol processor. Although decades have barely passed, it is transforming our concepts of information and calculation. Computers are thought to replace typewriters and desktops and filing cabinets. But the computer has yet to be understood for what it is of itself, we still view it from the impoverished model of what it replaces. McLuhan said that computers extend our central nervous system. Our CNS is not a symbol processor, it is a generator of personal realities. VR marks the end of the infancy of computation, the essence of the computer revolution is yet to come. Essentially computers are reality generators.
And reality is in the eye of the participant.