The KCOT teachers decided that all groups should have the opportunity to directly study the nitrogen and carbon cycles. The energy and water cycles were considered of lesser educational importance since these are cycles that are often studied in general biology courses. Therefore, the more educationally intensive treatments were assigned to the more ‘important’ cycles; nitrogen and carbon, with the energy and water cycles studied either in the traditional classroom, or as the no instruction control except for in the case of the two groups who created each of those virtual learning environments. The lesser value placed on the energy and water cycles can be clearly seen in Table 4, below, a matrix illustrating the instructional strategy for each of the four groups for each of the four wetlands cycles.

3.2.3. Procedure

The study lasted two weeks, 4 days a week on site at Kellogg Middle School. There were two 1.5 hour blocks per day of teaching time in the KCOT classrooms, from 9:00 - 11:30 and from 12:00 - 1:30. During each of these blocks, each of the four classrooms were held class simultaneously: one in virtual environment development and constructivist learning, one on traditional science, and two on other subjects currently being studied that were note wetlands-related.

For constructivist learning and world building, students had a total time-on-task of 6 hours, 3 hours per week. The same held true for each of the other cycles except for the no instruction control. All testing was conducted either before the project began or after completion of the project. This meant that almost all classroom time could be spent on-task.

The amount of time spent in each activity was the same for each of the four groups. Using the Carbon Group illustrated in Table 4 above as an example, their time was spent in the following manner:

• Constructivist classroom time: Self-directed study of the carbon cycle, designing and building the virtual Carbon World
• Traditional classroom time: Teacher-directed study of both the water and nitrogen cycles. No classroom time was spent studying the energy cycle.

Table 5, below, illustrates the amount of time spent in each treatment for each group. All blocks represent 1.5 hours of time.

3.2.3.1. Constructivist Instructional Program

During the first 1.5 hour constructivist block and half of the second 1.5 hour block, Carbon Group subjects had the opportunity to look over materials related to general wetlands ecology and the carbon cycle specifically. These materials were selected by the students from library guides. They could also view Internet-accessible information on wetlands ecology and their particular cycle, review CD-ROM and video-disk materials about the process, and develop an understanding of the concepts based on their experiences with these materials. There was no direct instruction in the constructivist classroom.

During the second half of the second 1.5 hour block, subjects met in groups of 10 led by a Human Interface Technology Laboratory (HIT Lab) Virtual Reality Roving Vehicle (VRRV) representative. In these groups, students assumed roles to make certain that all aspects of the design/build process would be covered, and to provide students with a sense of self-authority and responsibility. Roles were selected by the student. Two student per group were allowed to assume each role. These roles were:

• Project Facilitator (Project management and overall organization)
• Cybrarian (Documentation of the project, document management)
• Base Modellers (Creators of the base world in which all other objects were placed)
• Art Directors (In charge of composition and aesthetics of the environment)
• Behavior Designers (In charge of interactive components within the environment)

After roles were assumed, subjects discussed their view of what should be contained in a virtual wetland environment designed to portray the cycle that they were studying. Each group of 10 students designed their environment from the ground up. The design process included:

• Developing an understanding of the educational value of the environment and establishing learning objectives
• Defining the objects to be placed in the virtual environment
• Drawing sketches of the objects to be created on the computer
• Defining the behaviors, interactions and events inherent in the environment by creating a Behavior Matrix.

During the third and fourth 1.5 hour blocks each subject created their set of assigned objects on their classroom computers, either alone or in groups, using Swivel 3-D and a MacIntosh computer as their modeling tools. I constructed the final virtual environment from the students’ objects, drawings, and behavior matrices, on the Division ProVision 100, a proprietary system including both the programming/rendering language and virtual reality hardware.

The World-building Process

With any new technology, there is the need to foster new ways of working with the system. Regarding the development of educational virtual environments, a system to rapidly empower both the teachers and students needed to be developed, allowing them to be relatively self-sufficient and to use the technology to its best advantage.

The reasons for this were two-fold:

1. As Constructivists, the teachers and students needed to have as much hands-on control of the process as possible for their own sense of esteem, accomplishment, and empowerment.
2. The process had to progress without a HIT Lab staff member’s presence.

I began writing a teachers guide (Osberg, 1995a) during the summer prior to the VRRV project that was used in subsequent teacher training programs. This document, entitled A Teacher’s Guide to Developing Virtual Environments described the four-step process developed during previous and current world-building experiences with children (Bricken, 1991; Bricken & Byrne, 1992; Byrne, 1993, Osberg, 1993b).

Unfortunately, the publication was not available to the Kellogg Middle School teachers. Instead, HIT lab staff were present during the entire study to lead students through the world building process. By refining the process through the pilot study at Kellogg Middle School we were able to generate a much more comprehensive and helpful guide.

One aspect of the project that cannot be stressed enough is the need for effective project management. Building a virtual environment represents a departure from the way that classrooms are normally run even in the Constructivist environment at Kellogg. Students needed time to become proficient at creating 3-D models, to design and develop their ideas about the environment and interactions in the environment, and time to work together as a group. An example of the kind of time commitments required for these activities, based on the four-step process and our experience at Kellogg Middle School is presented in Table 6, below.

In this pilot study, the four regular classroom teachers took part in all aspects of Process Planning. They were in charge of logistics, schedules, selecting student groups and the wetland cycles to be studied. After these decisions had been made, HIT Lab staff took over and taught students the four-step world building process, starting with World Planning.

The Four-Step Process

The four steps or phases of world building are Planning, Building, Programming and Experiencing. By providing a progressive structure to the process, students had ample opportunity to think deeply about their constructions and to enhance their visio-spatial skills through drawings and models created both by hand and with the assistance of the computer. These four steps, as they relate to the Kellogg Middle School project, are described below.

During the Planning phase, KCOT teachers selected the wetland cycles as the content to be studied. Subsequently, students and teachers worked together to choose an appropriate educational theme relevant to the curriculum being studied. During the Process Planning portion of the project, teachers were expected to develop the following:

• Curriculum Plan, including:
  - Selected subject/concept area
  - Educational goal and sub-goal statements
• Assessment Plan
  - Content assessment tool information
• Process Plan, including:
  - Project Timeline
  - Logistical Description
  - Student Teams/Student Schedules
  - Responsibility Matrix for all involved staff
  - Contingency Plan (what to do when everything imaginable goes wrong)
  

Once Process Planning was completed, World Planning commenced. HIT Lab staff, teachers and students brainstormed about the curricular theme, and about information that could be visually and interactively conveyed through the virtual environment.

As is true in any design process, there are innumerable ways that one can choose to represent a subject, process, or interaction. Based on the triadic sign model, students were asked to evaluate their selections based on their value systems and on their previous experience with objects and relationships within a wetland. Discussions covered how and why students chose certain objects and interactions. Though sign theory or the triadic model was not presented in the classroom, students talked about their reasons for selecting certain objects within their groups. Certain objects were used as "window dressing" to create an analog to a real wetland. Other objects were used to reference real-world objects more obliquely. Part of the value of virtual environment design is to come up with virtual representations that have consensual meaning. It was interesting to note that some of the children’s inclusions were strongly cultural. For example, one student wanted to include a basketball as part of the carbon cycle, and another a tire representing pollution.

The World Plan was designed to assist teachers and students through the design process. It describes the environment as a whole (similar to developing a stage setting for a play), each individual object within that environment, who will be making it, how they will fit together and the behaviors in the environment, both object-to-object and participant-to-object.

The activities associated with developing a World Plan include, but are not limited to: brainstorming, developing thumbnail sketches, clay or other sculptural models and storyboards. Teachers and students were encouraged to get as deep an understanding about the environment as possible before breaking into individual groups to create their virtual objects on the computer. We found that having a picture of the environment as a whole and understanding its’ educational purpose enhances the experience portion of the project substantially. Students chose which types of models they wished to create. At minimum, thumbnail sketches in 3 views (top, side and crosssection) were required before getting on the computer.

In the first part of the World Planning process, the selected subject/concept was expanded upon by brainstorming about the kind of objects that might be included in the environment, how the environment itself may look and feel and how the students perceive that the educational goals can be met through developing a presentation in this medium.

The environment was discussed in terms of its functionality, educational purpose and content. Student discussions focused on the environment as a whole and on the part-to-whole relationships. In this way, students began to ‘ground’ their understanding about the wetland environment in a very personal way by evaluating their own beliefs, existing mental models about the wetland and by incorporating new knowledge into these models.

Sketching and modeling were also encouraged. Sufficient although limited time to talk and develop ideas within their groups was provided. Visual representations, metaphorical concept representation and stage setting and design were discussed.

The Object Matrix

The Object Matrix was the first document generated from the brainstorming and modeling session. In it, students wrote out the final object ‘list’, prioritized the objects on it and assigned responsibility to a team member for each objects’ creation. An example of an Object List taken from one of the Kellogg Water cycle wetland environments is illustrated in Table 7, below.

The Behavior Matrix

After the Object Matrix was completed, students began considering the behaviors or interactions associated with those objects. It is never too soon to consider the environment as a whole and how the participant and the environment will interact with one and other.

"Behaviors" are events that can be assigned to an object in an individual fashion (i.e. are inherent to the object, and are not ‘caused’ by the participant or another object), and those to which a clear cause-and- effect relationship can be established.

The behavioral characteristics that can be assigned to an object or object-interaction are listed in Table 8, below.

"Collidability" is the property of being able to sense when the bounding box of one object intersects with the bounding box of another. A bounding box is a spatial encapsulation device that ensconces and defines the outer coordinates of a particular object. Interactions between objects who’s’ bounding boxes intersect are often used to visually present ‘cause and effect’ relationships. As an example from Nitrogen World, children chose to make both the cloud and nitrogen molecule "collidable". When the bounding boxes of these two objects intersected, a rainstorm was ‘caused’ by that interaction.

Using the behaviors listed in Table 8 above, students began to think about what kinds of interactions or behaviors might be appropriate in their environment. From the brainstormed list, students developed the Behavior Matrix guide which was used as a programming aid when coding these behaviors during the Programming phase of the project.

Students learned that just because you CAN have an interaction doesn’t mean that you should. Interactions should add value to the experience and should be used judiciously. We suggested that they should, on paper, detail out each object and its’ potential interactions for evaluation purposes but that these interactions needed to be prioritized in terms of their educative and interactive value. Each environment was limited to 10 interactions per 5 minute block, due to the time limit per student and the number of interactions that could be completed within that time frame.

Another limitation faced by students was the use of textures. Due to limited texture memory in the Division computer system, students were limited to 3 textures per environment. This made it difficult to come up with environments that were very realistic. Most looked very ‘cartoony’, with bright blocks of color associated with individual objects.

After all the types of interactions and textures available had been discussed, the next step the students undertook was to develop the Behavior Matrix. This matrix was limited to a 10 by 10 set of interactions. This constrained the number of objects that could be considered for interactions and also limited the kind of interactions that would be available under certain circumstances.

Students had the opportunity to explore different representations rich in visual or auditory meaning, including those attached to interactions that the students selected. For example, if students chose to provide energy from the sun to the blue-green algae, it didn’t make sense to make the algae spin and change colors. However it might make sense to make the algae increase in size by either scaling or replicating it.

Interestingly enough, students immediately understood the concept of auditory feedback due to their experience with computers. Everyone understood the difference between sounds issued for errors, and those that connotated other actions. However, developing a sense of appropriate visual feedback was much more difficult for the children. In this respect, it was hard to get beyond the novelty factor of having the power to change almost any aspect of the representation at their command.

In Table 9 below is a partial illustration of the Behavior Matrix for the Water cycle used in Table 8, above.

Roughly translated, what is occurring above is happening at two levels: participant-to-object and object-to-object. During the participants’ initial contact with an object, an auditory cue tells the participant whether the object can be ‘grabbed’; all objects that are ‘grabbable’ have a higher pitched sound ‘ping’ attached to them. Objects that are not grabbable ‘gong’ the participant when touched.

When an object interacts with another object, for example the Sun’s Energy and the Lake, the product of that interaction appears. In this case, the interaction makes the water Vapor appear. The auditory cue provided during the interaction is a medium-pitched sound (bong), telling the participant that the interaction has occurred. The visual cue of the appearing water Vapor provides additional reinforcement.

Additions to the World Plan

We suggested that students and teachers attempt to document their design process as much as possible, by making sketches, models and writing pseudocode snippets for the programming portion of the process. Other media, such as video and still photography and newsletter articles can also be used to capture this process.

The Building component of creating a virtual environment is where the rubber hits the road, so to speak. Students learned that imagining is only half the battle. Building what you see in your mind’s eye is the other half of the relationship. The building process provided students with a means of making their imaginings concrete.

Object Construction

Students were encouraged to make paper-and-pencil sketches of their objects from three construction perspectives: top, side, and cross section views, as well as a combined view. These perspectives mapped to the Swivel 3-D software interface used to model the objects in 3-D. We found this type of analysis was helpful for simple objects and critical for complex, multi-component objects.

As an example, children found it relatively simple to make a straight cylinder. However, creating a virtual hand, comprised of many cylinders that must work in a certain fashion as a whole is a much more complex proposition. Though most students were quite capable even at the beginning of this phase of drawing (and modeling) simple objects, it took some tenacity and experimentation to figure out how to design and link objects to develop complex wholes. A dragonfly with independently manipulable wings is a good example of a complex object.

We found that asking students to mentally and physically take objects apart and reconstruct them in drawings from multiple perspectives was infinitely valuable to their understanding of the relationship between 2-D and 3-D representations. We were surprised at how their world view changed based on this activity. This was one of the most powerful aspects of virtual world design that we have encountered (Winn, 1995, 1997; Osberg, 1995b; 1997).

By visualizing objects from both the real and virtual environment in their mind’s eye and on paper, students begin to understand that part-to-whole relationship and its relevance to modeling and on a grander scale as well (Samuels & Samuels, 1975, Richardson, 1980; Richardson, 1969, 1994; Kaufman, 1984). Students’ language began to change. They spoke of whole-to-part relationships and on the value of multiple perspectives as a reflection of their world view. In addition, there is an amazing amount of power and latitude when it comes to describing and developing a "world". The sense of responsibility for developing an environment, especially a learning environment granted students an autonomous authority over their learning process that most had not experienced prior to this project.

Composition and Functionality

Another interesting component of the design process was the concept of composition and functionality. Composition of the world was discussed in terms of both aesthetic and functional appeal (Gigliotti, 1996). We used the metaphor of dressing a three-dimensional stage when talking to students about this issue (Laurel, 1991). Considerations included where the participant would enter, what they would have in their near and far environment, whether a clear path had been established, or whether the participant was intended to explore at will, and so on. We also presented composition with respect to the learning goals of the environment.

Scenes

One way to design the environment as a whole was to consider different ‘scenes’, similar to the scenes used in a play. These activities might tie to a particular region of the environment, or they may play out in what might be considered a common or multipurpose portion of the environment.

We asked the students to consider setting the scene from both the perspective of the participant and from a holistic view. Students had to consider how to keep the participant engaged in the environment, even if they followed a different route than what was anticipated.

In addition to setting the scene, students had to decide whether the experiences to be had in the environment would be scripted sequentially, or whether there was some flexibility in how a participant might experience the environment. These kinds of decisions were left up to the students, based on what they wanted participants to experience and accomplish in their environment.

Environment Size

We also asked students to consider the size of the environment. Because of the nature of the technology as it stands today, we found that we often have a difficult time controlling such things as ‘flying speed’ when an object is in a participants’ hand. Therefore, it is a more pleasant experience if the world itself is large enough so that all forms of movement can be accommodated (even a very quick mode of flying called turbo-fly), but small enough to avoid having to travel long distances.

Environment size is dictated by object scale. A good rule of thumb in designing objects in a modeling package is 1" = 1’.

Functionality

We found that if an environment is designed so that there are a set of activities that require a number of component parts, it is best to have those component parts relatively close to one and other in the environment. This eases navigational burden on the participant and also allows the programmer a visual check on the items required for the learning that the designers are attempting to facilitate. Of course, there are always those environments that are intended to be difficult, such as a treasure hunt world, or a puzzle world, or even some of the adventure games (where tools in the environment are intended to be misleading, or to be used at another point in the adventure). But for the most part, we have found that it is best to put things a) where participants can find them, and b) where net resultant ‘behaviors’ can also be viewed effectively. Students followed these design guidelines when creating their educational environments.

During the Planning phase, students described environmental interactions in their Behavior Matrix. This document was used to do the actual scripting of the behaviors.

We did the bulk of the scripting at the HIT Lab, since there really was no clear mechanism for allowing students to do the programming themselves. Two of the VRRV team members, Ari Hollander and Howard Rose, wrote a Supercard stack that provided students with a 10 x 10 matrix to determine interactions, and generated the code to implement them. Unfortunately, by the time it was completed Division had upgraded their software, making the Matrix obsolete. Instead, a paper matrix was constructed.

There were certain documents and files that we required the students to provide to us for the culmination of their design process. They include the following:

• Object List & Files
• Behavior Matrix
• World Description
• Final Object/Interaction Master
• Functionality Script, including learning objectives and how these objectives were to be met

After providing all of the above, students were rewarded with a completed environment which they could experience. The most difficult components of the world building process that we experienced were to get students to build a ‘base world’ upon which all of their objects would reside (if appropriate) and to fill out the Final Object/Interaction Master. An example of this document is presented in Table 10, below.

Additional Documentation

Students and teachers were also encouraged to provide any additional documentation they thought might help facilitate the process of creating the program files associated with the students’ virtual environment. At Kellogg Middle School, we were present for the entire process. However, all available documentation was used to create the final environments.

Depending on the complexity of the environment, the assembly process can generally take from one to several days. In the case of the Kellogg Middle School environments, it took a full 3 days to program the first two environments, water and energy. It took an additional day for each of the other two environments, carbon and nitrogen. Turnaround was very quick.

The environment in which students go through the Experiencing portion of the project can take many forms. Most of the decisions are based on the process selected for this portion-- is there post-testing involved? interviews? survey information to be collected? Are the students allowed to view each other going through their environment or are there constraints placed on the amount of interpersonal interaction that will take place?

At Kellogg Middle School, we worked on the process from both an ‘activities’ and a ‘logistics’ perspective. It was just as important to make sure we had enough power for the machines, tables and chairs in which students and evaluators could sit and pencils for survey sheets as it was to ensure that there was at least some opportunity for personal, private exploration.

Another issue was whether to capture students’ experience in some way, for example, on video tape, or with a camera. Lighting can become a concern in this case. This might also be an issue if the press has been invited to view students in their environment. Trying to film against glass during daylight hours is difficult at best. At Kellogg Middle School we did video-tape the students in their constructed environment, but we did so in the corner of the portable farthest from the windows.

During the last two days of the two-week program, students took part in the experiential portion of the 4-step process by experiencing both their own virtual environment, and an environment designed by another student group. The equipment used for the experiential portion of the project was the Division ProVision 100, a proprietary immersive system that includes a substantially enhanced 486 processor, wand, headset, and tracking system for both the participant’s head and hand. An illustration of the Division system is provided in Figure 3, below.

Figure 3 - Division ProVision 100 virtual reality system

The students were prompted by HIT Lab staff to "talk through" their experiences, and to describe what they were seeing, doing and experiencing while in the virtual environment. Occasionally, a student would ask what a particular object was, or what relationship it had to the cycle being experienced. This was especially true when a student was in an environment created by another student group. HIT Lab staff (or another student who was present) would often describe what the object was, or what it could be used for in that particular context. We felt it important to provide the students with as robust an experience as possible, especially as students had limited time (five minutes) in each environment, during which they were challenged to experience all that the environment had to offer.

3.2.3.2. Traditional Instructional Program

During the first 1.5 hour traditional block, subjects were guided by the traditional science teacher in reading wetland-specific portions of the textbook entitled Life Science: The Challenge of Discovery. Handouts provided the students with page numbers tied to the cycle being studied, a key word list, and a set of study questions to be answered in class during the discussion session. Students were told that they could seek out additional sources of information should they so desire. It should be noted that most students chose to stay in their classroom and work with the existing text.

During the second, third and fourth 1.5 hour blocks, students were given flowcharts and worksheets to fill out, with specific page numbers relating to the text. Both the flowcharts and worksheets had part of the cycle being studied was already drawn or described. Students were expected to "fill in the blanks."

At the end of the two-week project, some of the students (n = 41) got to experience a virtual world that they had studied using traditional learning strategies. The remainder of the students (n = 25) experienced a virtual environment in which they had had no instruction.

3.2.3.3. No Instruction Program

The No Instruction Program provided no direct instruction regarding a particular wetland cycle. The teacher who had the control group taught a completely unrelated subject during her time with the students. However, there was a great deal of overlap between the cycles studied and students already knew the water cycle in particular from earlier studies. Some also knew the energy cycle to a degree. Though no direct instruction was provided, it can be assumed that some of the information gleaned during general wetlands study or study of the other cycles may have provided useful information about the no-instruction cycle to the students.

3.3. Instrumentation and Testing

Instruments

The instruments used to test the meaning-making process were a hand-drawn concept map and a multiple-choice test designed by the KCOT teachers. These instruments are located in Appendix A, Quantitative Pre- and Post-Test, and Appendix B, Concept Map Pre- and Post-Test. The quantitative test was a 20-question test of all four wetland cycles studied. Questions were broken down as follows:

• 6 Carbon questions
• 5 Energy questions
• 5 Nitrogen questions
• 4 Water questions

The survey was developed by the VRRV team. This instrument was eventually used to test the attitudes and feelings of over 7000 school children who got to experience virtual reality, either through world-building or simply experiencing an environment. This survey is included as Appendix D.

In addition to the above instruments, subjects were observed while in the virtual environment. The subjects were then interviewed immediately after their experience to see what they remembered. These interviews were taped and a paper log was used to capture their recall of objects, interactions, concepts and processes while in the virtual world. An example of the log sheet is included as Appendix E.

Testing

The multiple-choice pre-test and the concept map pre-test were given to the students in their regular classrooms during the first period of the day on the Friday preceding the beginning of the world-building process. The post-tests were administered on Monday morning the week following the completion of the project.

Video, interview and survey information were collected either during or immediately after their virtual reality experience in both their created world and the world created by another group of students.

3.4. Expected Results

Anticipated results were based on the students’ existing knowledge of general wetlands ecology and of each specific cycle, coupled with the way each cycle was studied during the project. It was anticipated that the scores for all wetland cycles would rise regardless of instructional strategy, but that they would rise more for the constructivist approach than the traditional approach. Gains from the traditional approach were expected to be more substantive than those from the no instruction control. Moderate to low gains were expected in the no-instruction approach, because of transfer from other studied cycles.

The conclusions to be drawn from this exploratory study hinged on the development, or construction of meaning from the world building process. By using signs in a variety of formats and through the process of personal experimentation, evidence of abductive reasoning in addition to gains in student content knowledge were expected. The world building process is an intensive undertaking that requires deep understanding of a concept or process coupled with the skills to make that understanding manifest in a manner that can be experienced and enjoyed by others. It was expected this would lead to better post-test performance by subjects in particularly in the world-building group.

In addition, traditional instruction is better than no instruction whatsoever, and better post-test performance by suEbjects studying cycles using traditional strategies over those who did not directly study a particular cycle were expected.

Regarding the educational value of visiting a virtual world build by other children, moderate gains were expected. The use of this kind of technology has been found to be highly motivating, especially to middle-school age students (Winn, 1995). This project in particular had been presented as a high-stakes endeavor. Students were expected to be very much interested in both the technology in general, and in their personal environments specifically. However, their time ‘under the helmet’ was very short, so expectations about the value of the virtual environment experience were moderate at best.

The environment in which we housed the technology for the experiential portion of the study was a portable classroom in the back of the middle school. In addition to the short duration of their stay in each environment, the portable was also:

• Noisy (both the Division ProVision machines were loud, as were the twenty children present most of the time who were in the portable being used during the experiential portion of the study, in combination with the ten adults also present)
• Chaotic (we were conducting interviews at the same time subjects were filling out evaluations, and other subjects were ‘under the helmet’)

Based on these observations, moderate gains based on the virtual experience were expected. These factors can negatively affect learning in any classroom situation, since the potential for distraction was so very high. Though highly motivated, it was feared that students would not be able to concentrate in all the hub-bub.

Educational Emphasis

As mentioned above, the KCOT teachers chose to put less emphasis on the water and energy cycles, since these areas had been or would be covered during their standard biology class time. Therefore, educational emphasis was placed on the carbon and nitrogen cycles. For this reason, higher starting (pre-test) scores in both the water and energy cycles and lower pre-test scores for carbon and nitrogen were expected. In addition, greater gains in both carbon and nitrogen scores due to the extra educational emphasis places upon these two subjects were also expected.

Hypothesis and Scores

The highest quantitative and qualitative scores were expected to be associated with T1, followed by those associated with T2, then T3. In particular, improvement in concept map representations for the constructivist treatment which included virtual environment design were expected. The design process requires students to think deeply about the interrelationships between objects and interactions in the environment, which has the potential to lead to what is termed "high road" (Salomon, Perkins, & Globerson, 1991) or "deep" (Rose, 1995) transfer. However, we were asking the students to learn content, design skills and educational task analysis skills in conjunction with taking responsibility for a particular role within their group. As we were asking a great deal of the children in a very short time period, this could negatively impact their academic performance.

Regarding the second treatment (T2), children studied the cycle content using the traditional textbook/worksheet approach. Better scores on the traditional concept maps than those presented under the no-instruction treatment were expected, though it was thought that the maps would not be as detailed or complete as those designed during T1. Regarding T3, very moderate gains were expected, but certainly not as substantial as seen in T1 or T2.

Virtual Environment Experience

Higher gains in both quantitative and qualitative measures for the virtual environment designed and experienced by the student were expected, in comparison to the experienced environment designed by other students. However, the two "experiences" are clearly interrelated. It was expected that the experience of designing their own world would color the children’s experience of other students’ virtual environments, making it easier to assimilate information from the virtual space due to their design experience. It was assumed that they would try to grasp the same kinds of interrelationships in this environment as well as those designed in their own.

3.5. Data Analysis

The quantitative data collected via the multiple choice tests were analyzed using ANOVA with groups as a between-subjects factor and pre-post tests as a within-subjects factor. The data collected on the concept maps were rated by two separate raters for completeness, accuracy, and depth of understanding, by using a key-word and concept-flow identification system. The rating process is an extension of a holistic scoring technique developed at the Reading Center at the University of Illinois to assess children’s’ written compositions. Dr. William Winn and Dr. Patti Char at the University of Washington have piloted an extension of this technique to evaluate pictorial representations, such as those found in concept maps. A first pass was made through the data to seek out representative "anchors"; the best and worst representations that the raters agree upon. A second pass was made through the data during which each rater evaluated representations or descriptions holistically based on the anchors. Differences of opinion as to the value of a particular item were negotiated on a case-by-case basis. The same technique was used to analyze the performance log.

The multiple choice test provided a measure of declarative knowledge and the concept maps a measure of interrelational knowledge. The interview sheets provided perspective on the children’s procedural knowledge regarding the subject area as they described in narrative form their personal interaction with the information presented in the virtual environment.

The survey data was a 15-question Likert Scale and essay question document that was administered to the students after their interview was complete. Students were asked to judge their perceptions of their enjoyment of the process and experience, the educational value of the process and their experience and whether they would choose to undertake such an activity again.