CHAPTER 6

TESTING MY COLLABORATIVE WORLD

Having built my world, I wanted to run a pilot test which would test marbles world against my stated objectives. The pilot tests would be somewhat informal, yet give me some experience with running tests should a more formal experiment become warranted. I would ask the participants various questions about their interest in the world, get feedback about the technical aspects of the world itself, and ask about their desire to return to participate again. At the same time, I also wanted to obtain data from the participants that would hopefully support or refute a simple hypothesis. I considered a few possible pilot test goals such as comparing performance between anonymous participants and participants who met each other beforehand or comparing collaborative behaviors of participants before and after participating in the world. The architecture of the world, being so easily extended, opened a long list of feasible pilot tests I could run. The world could be modified with minimal effort to run many different pilot tests.

I decided to run a pilot test which would grapple with a question related to one I had read often in magazine articles about Virtual Communities: “what makes people remain in and return to a Web site?” Many articles propose that people like to visit Web sites where they feel like they are part of a community. The community can be as simple as those who drink the same cola, or as life enhancing as people who are fighting big government against a perceived wrong. The idea that people come to a virtual place where they feel comfortable with the other people who are there parallels a real world

phenomenon that keeps people coming back to their church or university. I wanted to take the comfort idea in a slightly different direction and consider whether people remain in and return to Web sites because they have a say in what takes place at that Web site. Howard Rheingold’s Electronic Minds Web site was an example of a text-based Web site where people visited and drove the conversation themselves. I visited Electronic Minds and opened a discussion on whether the analogy of an Information Superhighway is a good one for explaining the Web to the masses. I certainly was drawn to Electronic Minds because I had a say in what happened there. I quickly perceived myself as being a part of a community with the others visiting.

In terms of internet based 3D environments such as my marbles world, allowing for audience participation is a newer frontier. I imagine a sandbox world where a participant can connect via the Web, bring his or her own visual objects into the world, and share an experience with others using the objects each participant brings. Or, I can imagine a gallery world where participants bring their own works of art into the world and work together to present them in the most attractive and functional manner. Gallery world’s gallery could even be built from scratch by a participant base given appropriate tools. Or, I an imagine a virtual golf world where participants build a golf course before playing golf.

These three imagined worlds can all be feasibly built using VRML, an external authoring interface and a Java server. The worlds I suggest in the last paragraph each have different levels of participation. A sandbox world could have virtually no pre-set objective outside of sharing time with others. A gallery world might be quite a bit more organized. A golf world might follow very specific rules for playing golf. The questions that begs research are “how much freedom to Web participants want?” and “are Web participants more apt to stay in and return to Web sites where they have control over what takes place in the world?”

The first question is very interesting, but I believe depends too much on individual preference to test with a limited subject group. The second question is interesting as well, but I think may have too broad a scope for a simple pilot test to provide meaningful results. The question I decided to investigate is whether Web participants perform better in worlds where they design the world and participate in the rules determination than those Web participants who just show up, read the rules, and participate in a pre-built, 3D world.

Since marbles world provides a library of objects, a library of effects, and a library of rules from which to define what takes place in the world, there is a opportunity to set up a pilot test between two groups: one group which together defines the rules, builds the board, and runs the simulation, and another group which just runs the simulation the previous group defines.

The hypothesis I offer is that the group that creates the world through its participants will perform better given the rules they choose than the group that just follows the rules and design set by the previous group. I am not sure of the full implications of those expected results, but I believe they are related to experiments run on a participant’s sense of immersion in 3D worlds. Those experiments suggest that virtual participants feel more immersed in a world when they are interacting with it based on their ideas and actions. If the hypothesis holds true, Web designers would be supported in creating Web sites with more flexibility. If the hypothesis is negated, Web designers may well be justified in continuing to mass produce more static Web pages designed to work well with even the first Web browsers and Web TV.

It seems to me that the Web becomes significantly less important if Web participants are not craving participation with others. Broadcast TV can deliver content to the masses effectively. The Web provides a possible direct connection between any two computers (or people) in the world. With the Web, there exists an opportunity to use technology in completely new ways. If people can use a Web connection to create something new of value, proliferation of the Web may be quite a wise investment. Since my hope through this project has been to become effective at using a technology that provides new and useful ways of collaborating electronically, this pilot test will also provide some initial feedback as to whether people would use the technology given the chance.

As a last paragraph before outlining my pilot test, I want to mention how communicating and sharing through a Web site could dramatically change the way certain things happen in society. Today, many of the products and services consumers purchase in the marketplace are first introduced through a product development cycle which very often takes place within the walls of large corporations. Consumers get the opportunity to provide feedback to product development personnel through surveys and focus groups. Although the Web has made it easier for consumers to provide feedback, flexible Web sites could allow consumers to design their own products and then take them directly to a product or service provider for realization. Similarly, many games are created by game development companies which instead could be creatively developed and tested within the confines of a flexible Web site by independent Web participants.

Product and game development worlds can be built along the same design philosophies I follow in building marbles world. The question is whether people would use them given the chance.

Preparing Marbles World for a Pilot Test

In order to prepare marbles world for the pilot tests I wanted to run, I decided to remove some of the code which had provided flexibility. I figured the results would be too hard to analyze if participants could choose from the full rules set I had created and tested. How could I compare simulations run in competitive mode and collaborative mode? I decided to limit the pilot tests to collaborative mode since I was most interested in collaboration through the Web anyway. I then decided to use a fixed set of objects, events and physics to cut down on the learning curve for participants. By discussing the upcoming tests with fellow colleagues, I came up with a fixed way to manage the slant queue. I decided the world would load initially with the first two slants already determined. I found that the group design activity was more interesting with the two slants already in the queue. Lastly, I spent time fixing the server code so that a specific list of questions would be asked to any group that was determining the rules. I then was able to focus my code test plan more specifically.

My Pilot Test Plan

I will find twelve or sixteen subjects who are willing to spend a couple of hours maximum in front of a computer monitor participating in a virtual world. Three or four subjects will participate at a time. For each group, I will give them written instructions to read and then spend time with them answering all there questions except questions about group strategy. Each group will participate in two simulations. The first time around, the group will collaborate in a world designed by a previous group. The second time they will choose the rules and physics for the world, design the world, and then run the simulation in a manner similar to the first time. All participants will work toward an objective of reaching a fixed number of goals in as little time as possible. After each simulation, all participants will answer a post-simulation questionnaire. I will then look at the differences between answers from design round participants and non-design round participants.

All participants will be able to read about the world before they participate and ask questions about any directions or procedures they do not completely understand. But, the participants will have no previous physical practice with the world. I will only require that participants have average or better mouse skills. My world will be fixed in collaborative mode in order that a group’s success will depend on how quickly the participants get at least one marble (but any marble) to reach all the goals that appear in the world. The world will keep track of how many frames it takes to finish the simulation, as it also keeps track of the rules and design. Should the participants be unable to reach all of the goals within 2000 frames, I will stop the simulation and count the number of goals they were able to reach.

After their participation, I will ask each participant questions that rate their experience in the world as to how much they enjoyed the experience, how well they believe the technology performed, how likely they would return to the world, and how immersed they felt in the experience. The simulation will be considered more successful if participants enjoyed the experience, supported the technology, felt likely to return, and felt immersed in the simulation. My quantitative data will report on the performance of the groups. I will introduce a scoring system to the world that will score points for getting to goals in the shortest time possible. I will compare the cumulative scores between the design simulations and the non-design simulations as a more objective comparison of the two groups in action.

The Post-Simulation Questionnaire

The questionnaire I will give to each participant after they participate appears as follows:

Please answer the following questions on a scale of 1 to 7, 1 meaning you strongly disagree and 7 meaning you strongly agree:

1. I enjoyed participating in the simulation.

2. I became immersed in the simulation with little awareness of other things going on around me.

3. I would like to participate again with the same group of participants.

4. I would like to participate again with a new group of participants.

5. I thought the technology worked well.

6. The interface was natural to use.

7. I felt like I was treated as an equal in the simulation.

8. I learned something about collaboration during the simulation.

9. I had more control of what took place than I anticipated.

10. I would like to have more control of what happens in simulation.

Please provide written comments related to the following:

11. Please describe your frame of mind when you started the simulation.

12. Please describe your frame of mind during the simulation.

13. Please describe your frame of mind right now.

14. Please list any frustrations you experienced while participating in the simulation.

I ask question 1 because I believe Web visitors must enjoy participating in a world if they are to become part of its community for the long-term. To evaluate my world, I must determine whether participants enjoyed the simulation. I ask question 2 because I am interested whether my world was immersive or not. Immersive worlds are more successful than non-immersive worlds in keeping a community together and vibrant. I ask questions 3 and 4 since more successful communities have members who want to return to visit often. I ask question 5 to differentiate between disinterest from problems with the technology and the disinterest from the underlying idea for the community. I believe the technical problems can be easily overcome. Fundamental idea problems would be more difficult to overcome. I ask question 6 to grade myself on my interface design. I ask question 7 to confirm that a server is a fair and just facilitator of actions. I ask question 8 hopeful that participants will learn about collaboration during their participation. I believe they would learn even more if they participated in a competitive mode as well. I am not providing competitive rounds as part of the pilot test. I ask questions 9 to get a sense of whether expectations from reading my instructions coincide with the reality of the simulation. I ask question 10 to see how participants feel about not having as much control as many video games provide. I ask questions 11, 12, and 13 to get a sense of how participants’ thoughts change throughout the pilot test. I ask question 14 to get feedback that could help me make the simulation better. I believe each question will provide valid feedback. I now run the pilot test and review the results in Chapter 7.