Evaluating the Production of EVE and Evaluating the Aspects of EVE By Mike Burton (burton@halcyon.com) and Gary Withrow (gwithrow@ncr-mpd.ftcollins.ncr.com) Project for CMSC 828S: Virtual Reality, Telepresence and Beyond University of Maryland Professor Ben Shneiderman Table Of Contents A.1 - Abstract A.2 - EVE Production Introduction A.3 - Survey Analysis A.4 - EVE Production Summary B. Evaluating the Aspects of EVE B.1 Introduction B.2 Motivation for a Well-Working EVE B.3 Message to all Reviewers Requested to Evaluate EVE B.3.1 Background Information to All Users B.3.2 The Survey B.3.3 Voluntary Participation B.4 The Sources of EVE Users B.5 Evaluating the Users of EVE B.6 Individual Survey Questions (1-9) B.6.1 Question #1 B.6.2 Question #2 B.6.3 Question #3 B.6.4 Question #4 B.6.5 Question #5 B.6.6 Question #6 B.6.7 Question #7 B.6.8 Question #8 B.6.9 Question #9 B.7 Proposed Improvements to EVE B.8 Conclusions B.9 Difficulties in Evaluating EVE C. - Bibliography Appendix A - EVE Production Survey and Results Appendix B - Actual Survey Responses from Evaluating EVE Appendix C - Credits (Required for grading credit) A.1 - Abstract In the Fall term of 1993, Ben Shneiderman of the University of Maryland presented a course titled "Virtual Reality, Telepresence, and Beyond." The course was offered to students at the University of Maryland and to remote students in 12 states across the United States. The remote students viewed the class through live video feeds and via satellite with a 2-5 day tape delay through National Technological University. The first project for the course was called EVE which stands for Encyclopedia of Virtual Environments. EVE is a series of short articles authored by members of the class. Originally, EVE articles were envisioned to be written by pairs of students taking the class from different locations. Real world difficulties such as students dropping the course and getting pairs of authors at different locations to sign up for each EVE topic forced the articles to be written by both single authors and by co-located pairs of authors as well as pairs of authors at different locations. The topic of this paper is to evaluate the EVE class project from two perspectives. The first is in the evaluation of the production of EVE articles. This will be examined as a case study in Computer Supported Collaborative Work (CSCW). (This term also appears as Computer Supported Cooperative Work.) The second is the evaluation of EVE itself as an encyclopedia for users to extract information in a quick and usable way. A variety of users completed a survey to evaluate the aspects of EVE as a tool. Recommendations for improvements to the EVE class project are then given based on the findings. Both members of this project are remote students living in Seattle, Washington and Colorado Springs, Colorado. A.2 - EVE Production Introduction A survey distributed electronically to all the EVE authors was used to evaluate the production of EVE. The results of the survey are included so that the reader can gain insight into the authors, evaluate our conclusions, and potentially use these results for some other analysis. There are many good suggestions relating to the administration of a class project such as this. Although these will not be of interest to everyone, some may help make this an even better class in the future, so the data was kept. The data summary is quite sizable, so it is attached at the end of the article. In evaluating this data, it is important to realize that the authors of EVE are enrolled in a graduate course in Computer Science so communicating via Electronic Mail (E-mail) and the general use of computers should be very familiar to them. Many more problems could potentially arise trying to perform the same tasks with subjects from the general public. The first evaluation will be qualitative and not quantitative. The number of teams that experienced problems is not as important as the fact that some did. If we tried to examine small software projects, we would probably not see many cases that called out the need for configuration control, properly commented code, or many other commonly accepted software engineering practices. Similarly, since individual EVE articles were small, the responses will be examined for problems experienced in CSCW and solutions and suggestions offered by the subjects. A.3 - Survey Analysis Question 1: If you worked with a student not at your same location, what problems were presented by working with remote students? The very first question in the survey brought out many problems. A common complaint was that because of the long turnaround time between sending a message and getting a response, use of E-mail was frustratingly slow, especially when trying to get organized and make decisions. Another was that when the E-mail is sent out, the sender was not sure it was received. Also, some people felt that the recipient was less accountable to E-mail and may not have acted as quickly as they would have if the message had been delivered directly. These problems were mentioned in the book Connections [1] as well as being commonly mentioned in CSCW writings. These problems should be considered all the more serious when you remember the backgrounds of the people having the problems. Other problems mentioned in the first question were: 1) differences in time zones complicated other forms of communication 2) frustration with the smaller information bandwidth of E-mail, specifically the respondent could not sketch ideas, but also lacking in E-mail is the capability of visual feedback and the use of gestures to communicate and 3) lack of version control. In this case, two partners often were not working from the same version of the article. Interestingly enough, these are common problems encountered in CSCW and work is being done to try and address all of these problems. (Differences in time zones explicitly can not be addressed, but tools for groups working in different locations and at different times are being worked on.) Appropriate tools would include extended E-mail with feedback when messages are received, decision support tools, group editors, conferencing and sketchpad systems to expand on the E-mail bandwidth and distributed document and comment control systems. In the CSCW video shown in class, Marilyn Mantei showed Clearview, a system that allowed drawing on the screen and visibility of the other person you were interacting with through the screen. This answered the E-mail bandwidth problem as gestures and facial expressions were visible as was a shared workspace. She also demonstrated three tools SASSE, CAVEDraw and VooDoo that could all be used together. SASSE was a group editor, CaveDraw a group drawing program and SASSE supported a virtual office. With SASSE, video was available to again answer the ASCII text bandwidth problem while group editing and drawing were also supported. Many other projects are in work to answer some of these problems including VideoWindow [2] that provides a shared video space through a very wide field of view video system that supports two remote facilities relating as if only separated by a window, SIBYL [3] a group decision support system that allows graphical representation of decisions and supporting and detracting arguments and alternatives, PREP editor [4] which provides version and comment control along with the work plan, and ClearBoard [5] which is either the same thing Marilyn Mantei Clearview, or a very similar effort. Many of these systems are very research oriented and will take much refinement before they can be turned into a commercial product, but the PREP editor concept would have greatly eased many of the problems the class had with EVE. PREP displays three columns, the left hand column is the Plan column that has the plans, objectives and goals of the project and tools to help develop and maintain these. The middle column is the text of the document, and the third column is comments. This appears to be a very convenient method to make comments and have them tied to the original text, but not have the original text affected until the person responsible reviews the comments and applies them to the text. It also appears to be very easy to then go through your documents and have the comments beside the appropriate text while you are working incorporating any changes. Also, if used properly, it may be easy for the person who made the comments to go back and review the status of their comments. (The need for this capability is called out below.) Question 2: Was getting your group organized and deciding on the topic difficult? What made it easy, or could have been done to make it easier? Question 3: Was coming up with an original outline and starting the collaborative work of the body of the project difficult? Questions two and three were intended as follow-ups for the first question to probe for further evidence of problems coordinating work via E-mail. The answers to the first question were well thought out and respondents made every effort to be helpful and informative, so questions two and three were not required in this case and did not bring out any problems not already covered. Question 4: What methods did you find to integrate the sections of your work? Question four was intended to bring out the techniques used by the pairs to coordinate their work and brought out two solutions. Either one person wrote a draft that was iteratively expanded upon until both were satisfied, or the article was divided into parts and the authors wrote their parts and then iteratively commented on the others work and incorporated comments into their own sections. These techniques worked well for most groups when the number of contributors was two, but version control and keeping track of comments from multiple reviewers would have complicated this situation. As mentioned earlier, work is being done in this area. Question 5: What methods did you find to incorporate others comments? Question five was aimed at finding out how people incorporated comments. The result was very interesting. There were many variations on how comments were incorporated, but each thought matter-of-factly that there method was probably the common sense approach. Responses ranged from either incorporated them or ignored them depending on how the recipient felt, to sending a thank you for the comments and incorporating the ones they agreed with, to implementing the ones they agreed with, trying to persuade otherwise for the ones they did not agree with, compromising if the persuasion did not work, and finally still ignoring a comment if the previous attempts did not work out. It is interesting to note that at least one respondent appreciated the fact that E-mail can seem cold and impersonal and made sure they thanked the person for the comments. In E-mail, the positive aspects are often assumed by the writer, but not always by the recipient which can lead to communications problems. This was a very creative and effective approach. Another very good observation brought out by this question is that the teams could only allow one person to modify the document at a time. A document and comment control system could have been a helpful here, and all the much more on a larger project. In some corporate settings, a document can get sent out for review to many reviewers, and each comment may be tracked for incorporation or justification for not getting incorporated. Obviously, the methods mentioned here would not be sufficient for such an environment. Again, employing a computer to help with version control and to organize comments is desirable and being worked on. Question 6: Did you find yourself compromising more or less than you would have working face-to-face? Question six explored the possibility of people compromising more or less than they normally would have to accommodate the E-mail environment and brought out some interesting thoughts. Some respondents said they compromised more because of time constraints or that it was too difficult to explain their position. Others said they compromised less because they could not afford to given the time constraints and since the E-mail was less personal, it was easier to not compromise than it would have been face-to- face. Although changing one's natural reactions was originally assumed to be negative, one responded that they had more time to reflect on the other person's suggestions resulting in them compromising more than they would have face-to-face, but the article was better because of that. Question 7: Did you experience any mis-communication because of no face-to-face contact? Question seven dealt with mis-communication. This had various responses too. Some said they had very little communication, it was just easier to divide things up and work on their own. They realized communication was difficult and minimized the necessity for it. Some thought that this lessened the potential quality of the product, but it worked given the circumstances. A comment also repeated the aforementioned problem of working from different versions of the article. Comments were made that the tone of their message was misunderstood or that their remarks were taken as criticism. Feedback of seeing facial expressions could have avoided these problems. Question 8: Did you experience any flaming, or people being overly cautious not to flame because of the lack of face-to-face communication? Question eight dealt with either excessively violent verbal reactions called flaming, or if groups felt inhibited in their communications knowing that flaming is possible. No flaming occurred during the EVE project. Some felt slightly inhibited making sure this did not happen, and one response was strongly inhibited and thought it may have hurt the quality of the work. In their words, "I felt very uncomfortable commenting on someone else's writing and not being able to get any feedback on how it was being accepted, so yes, I was very cautious to repeatedly tell them it was there decision, these are only suggestions, and other statements like that. I also let small things go that I would have suggested about had we been meeting face to face and things been going well." Question 9: How would you rate your satisfaction with the final result compared to if you had done it yourself? Question nine was to get a feel for people's satisfaction with the experience and was not aimed at discovering new problems or techniques. Feelings on this question were very mixed. Many thought that they could have done better working by themselves. Some thought maybe they could have written better, or that it would have flowed better, but appreciated the extra point of view and help the partner gave. Others were clearly positive and think the resulting article was clearly better. Question 10: How would you rate your satisfaction with the final result compared to if you had done it in a small group meeting face-to-face? Question 10 asked if people thought the project could have been done better with face-to-face meetings. Many reacted that face-to- face would have made decisions and organization easier, but that the writing would still be divided up and worked on individually, so as a whole, they were satisfied with the E-mail approach. Question 11: Did the project take longer, shorter, or about the amount of time that you had anticipated? Question 11 was just to again sense the level of frustration people might have had. Most people thought the project took a little less time than they had anticipated, although for some it took a little longer. One person thought that there was a gain in efficiency because each partner worked on the parts they were most comfortable with first and they did not lose time struggling over hard areas. In most cases, they did not have the same hard areas, so the work was faster. Question 12: We are interested in any other thoughts you may like to share about your work on the EVE project. Question 12 was a chance to make any last comments. I will leave this section for you to read in the survey. Many good ideas were expressed. A.4 - EVE Production Summary Overall, EVE was a very successful class project in teaching people about Virtual Reality topics, giving the students first hand experience with CSCW, and produced an impressive product. Many common CSCW problems were encountered, but the class was full of bright, ambitious individuals that overcame the obstacles and made the project work. Our thanks go out to all the EVE authors that sent in replies to the survey and the speed with which they replied. Many took great effort to be as helpful as they could. Some that wrote articles by themselves, but coordinated sections provided examples from those experiences even though they could have just said "N/A". Again, all the effort is appreciated. B. Evaluating the Aspects of EVE B.1 Introduction The second survey taken was in the evaluation of EVE itself as a VR tool. EVE was intended both as a virtual reality information source tool for anyone wanting to learn about or research areas in Virtual Reality (VR), and as a tool to aid undergraduate students in computer science related studies to both learn about the aspects of VR and about which classes/courses to take in order to pursue work in VR. This second survey was intended to provide individual feedback from the actual users of EVE to determine the useability of EVE and if EVE had achieved its goals for the undergraduate CS students. B.2 Motivation for a Well-Working EVE EVE is just one of many encyclopedias that has been developed as an on-line resource for students and others who wish to access information electronically. Since 1983, The Interactive Encyclopedia System (TIES) has been in development to allow users to easily traverse a database of articles by just pointing at certain highlighted words of interest. Since then, many different databases of articles have been developed for users to explore information resources in an easy and appealing manner [6]. Although most of the evaluations of EVE have been from users of pure text (ASCII) versions of EVE, it has been the attempt to utilize this information to make EVE a more attractive resource than it already is. By having a resource available electronically, like EVE, users from all around the world can tap into the information that awaits them on the fast, growing, and exciting topic of virtual reality. B.3 Message to all Reviewers Requested to Evaluate EVE Although the Hypertext version of EVE is available from anywhere in the world (with internet access), most of the evaluations had come from those who had used EVE in the ASCII, or raw text, format. The following message was posted to several different virtual reality newsgroups. The message was intended to provide enough information to the user on EVE, but not too much that they would feel overwhelmed when reading it: B.3.1 Background Information to All Users In the Fall term of 1993, Dr. Ben Shneiderman presented a course titled "Virtual Reality, Telepresence, and Beyond" shown throughout the U.S.A. from the University of Maryland. A large part of the course was an electronic class collaboration on the construction of EVE. EVE is an Encyclopedia of Virtual Environments. EVE is a series of short articles authored by the groups of members of the class. It is available in ASCII and on some hypertext platforms. As a follow-up effort, students Gary Withrow and Mike Burton are evaluating the results of the project. Part of this evaluation is to get reactions and opinions of people who read and use EVE by answering some survey questions. In order to gain access to EVE in ASCII, ftp to 'umd5.umd.edu' login as 'anonymous'. The directory is 'pub/eve' There is one file containing all of EVE called 'eve.ascii' and the individual articles in EVE have extensions .txt *Also* included is the below survey called 'survey.txt' Your response is completely voluntary, and you are certainly welcome to read and use EVE without responding to our survey. The survey will aid in improvements to EVE and electronic collaboration studies. Thank you very much for your time and for sharing your EVE opinions with us. These results will be anonymous, so please feel free in expressing any reactions you may have. The answers can be as long or as short as you like. Please send your survey responses to: "email@...etc." B.3.2 The Survey ------------------------------------------------------------ Evaluator information: The following will help us to identify trends in the answers based on the evaluators background. Providing this information is completely optional, but strongly appreciated. Sex: Age: # years of computer science / engineering experience: ------------------------------------------------------------- **SURVEY** 1) What are your overall impressions of EVE? 2) What is your impression of the consistency between the articles? 3) Many articles were produced by pairs of authors using email. What is your impression of the consistency within articles? 4) How do you rate the readability of the articles? 5) How do you rate the technical/information content of the articles? 6) EVE was targeted for undergraduate CS students. Do you think that the writing was appropriate for this level? 7) EVE was partly intended to provide information to undergraduate students on the classes to pursue in order to equip the reader with abilities relating to Virtual Reality. Do you think the recommendations were appropriate? Too many? Too Few? 8) Any other strong points to EVE? 9) Any other weaknesses that stand out? B.3.3 Voluntary Participation The initial information above explicitly informed the reader that they were welcome to read and use EVE on a completely voluntary basis. They were in no way forced to complete a survey, or bribed in any way. All respondents had participated on their own will. B.4 The Sources of EVE Users As mentioned above, EVE was posted on internet newsgroups, made available at both Gary's and Mike's places of employment, and to all the actual authors of EVE. The surprising fact was that we did not receive any evaluations from the internet(newsgroups) users of EVE. Perhaps, many uploaded EVE to their own systems, but just did not have the time to fill out a survey. We had received one dud mail message from a user who could not send mail to us because of the type of specialized internal work mail they were using. One of the distressing facts was that one user replied from the United Kingdom who said that many people do not have the ftp utility and were unable to get a copy of EVE. This individual had requested that we post EVE itself on the newsgroup, but because of the size of EVE, we did not want to swamp the internet with a large text file. About 30% of the responses came from classmates, or other authors of EVE, and the remaining responses came from work associates and friends. The users who did not have access to EVE electronically were provided a clean, readable hardcopy. There were form-feeds between the different articles and table of contents in the front in order to allow the user as easy access to each article as possible. Only three users used this method and responded with handwritten surveys. These are separated from the electronic responses in the Appendix B section. B.5 Evaluating the Users of EVE The following data was taken from voluntary information provided by the users. In no way is it used to attempt to identify individuals to answers, but only to identify trends in answers. (no significant order) Ages: 32 22 37 25 26 34 27 26 40 28 25 22 44 43 28 25 The average age for reading the article was 30. Excluding the top two and bottom two ages gives us an average age of 29. We did not find an correlation of answers between male/female respondents, so this information was not used and is not listed. (Only one female evaluated EVE) The average amount of computer experience that the users had was 9.9 years. About half of the users had 5 years or less of computer experience while the other half had 10 years or more. (no significant order) Years of Experience: 9 10 2 15 13 13 16 5 3.5 14 5 3.5 4.5 20 20 5 B.6 Individual Survey Questions (1-9) B.6.1 Question #1 What are your overall impressions of EVE? The responses were overall very similar. One response to sum them up was "It was very informative, well written, and well organized." Most of the individuals felt that EVE was impressive both as an educational and entertaining source. The main addition that the users would like to see in EVE is a general overview of VR. Apparently, each article has its own specific introduction, but an overall good introduction appeared that it might improve its use for beginners in the Virtual Reality system. One user replied that some images in the text version as well as more images in the hypertext version would be helpful. B.6.2 Question #2 What is your impression of the consistency between the articles? This answer was basically either a good consistency or not good consistency. Many of the first time users thought that it had good consistency, but mostly based on the fact that there was a diversity in the authors themselves. One individual thought that there was some redundancy between the articles, but another thought that they was just enough over-lapping of articles that would allow them to stand alone outside EVE (as a specific single source of information). Those that felt there were inconsistencies had similar reasons that some articles had high level easy reading material while others had too much detail that was difficult to follow. Consistent overviews for each article was also not seen. Some articles had good overviews while others might jump right into the "heavy stuff". The formatting was seen to be very consistent in both versions of EVE. As we know, this should be credited to the class members who assembled EVE as a single unit. A good suggestion from one user was that it might have been good to have a master proofreader be assigned to increase the consistency. It seems that in this case, this individual assigned would have to be a highly efficient email user and fast reader in order to coordinate consistency with such a large number of users and large written work. B.6.3 Question #3 Many articles were produced by pairs of authors using email. What is your impression of the consistency within articles? All respondents felt that there was good consistency within the articles, regardless of articles written by pairs or by individuals. B.6.4 Question #4 How do you rate the readability of the articles? Readability is simply defined as interesting and able to read easily. The respondents answered this question well and to the point. Almost all the respondents felt that the articles were very readable. Many indicated that the articles they read were enjoyable, understandable, and interesting. Because of the difference in the articles chosen to read are naturally going to be different from user to user, the answers did show some effect of this. Several did feel that as most of the articles they read were readable, there were some articles that are a "little wordy" or have "too much detail". This is a very subjective question and can have a high dependence on what the interests of the reader are. For instance, if one reader A has interests in sound synthesis while another reader B has interests in psychology, and the article read was on Realism vs. Abstraction, reader A may feel the article was wordy while reader B may feel the article had just the right amount of information. B.6.5 Question #5 How do you rate the technical/information content of the articles? There was mixed reaction of the users to this question. Some users felt that about half the articles had an adequate amount of technical information while other articles went into too much depth. Other users felt that some of the articles could have more information. Overall, on the amount of technical content, 30% felt there was a proper amount, 20% felt there was too much, 25% felt there was not enough, and 25% of the answers were inconclusive. I believe that this variation shows there certainly is a difference in the amount of difference in the technical content in the articles, but also there is most likely a difference in the reading level of the users. The statistics showed that the average number of years of computer experience was 9.9 years, but this may not actually reflect the reading level. One very interesting response was that "it is said that for every equation used in your article the writer will loose about 10% of their readers." B.6.6 Question #6 EVE was targeted for undergraduate CS students. Do you think that the writing was appropriate for this level? 75% of the answers reflected a favorable response to this question. The remaining 25% had mixed responses. Some of these felt the articles could have been more meaty while others felt that there were terms used that made it less understandable. This last response was from an evaluator who did not have access to the hypertext version of EVE. Using the hypertext version could allow easy access to the glossary of complex terms at the end of EVE. B.6.7 Question #7 EVE was partly intended to provide information to undergraduate students on the classes to pursue in order to equip the reader with abilities relating to Virtual Reality. Do you think the recommendations were appropriate? Too many? Too Few? This question prompted the most reaction from the users. The individuals had different ways of presenting their view, but overall, there seemed to be an inconsistency in the manner with which recommendations were made. Many felt that there were simply too few recommendations. Many articles were seen as not having any recommendations at all. Some readers asked for more detail of subjects within classes as some recommendations were generic. One respondent said 'suggestions like "study lots or math and science" aren't going to help much'. Overall, most felt there were too few quality recommendations in the articles and there was much inconsistency between the articles when recommendations were made. B.6.8 Question #8 Any other strong points to EVE? The major benefits pointed out by the users were: the logical organization of the sections and subsections was good, there is good information on all areas of VR, EVE is accessible from anywhere in the world (with internet access), it is interesting, it is good reference for people interested in VR and for more information in certain subjects (using the bibliography), and EVE is simply a good introduction to those newcomers in VR. A lively response that sums it all up was EVE is "Well written!! Highly enjoyable!! Very informative!!" B.6.9 Question #9 Any other weaknesses that stand out? The responses to this question are ones that should not be taken lightly. As our users have evaluated EVE, the improvements they point out have the potential to certainly enhance the already powerful tool of EVE. The main weaknesses that have been pointed out are: EVE could use an abstract/overview. Many students just learning about VR could use a basic overview of both VR as a topic and EVE itself as a document. Part of the information I had provided to the reader during the survey was some background information on how EVE was created and what it was about. This is done in the credits section that Mike Dennehy had written, but naturally, the credits section does not provide an overview of what VR actually is. EVE needs an improved method of indexing to the article of choice. (This request came from an ASCII user of EVE). Many users may only be able to use EVE with an editor (like 'vi' on a UNIX system), and will need to have specific instructions on how to search down to their article of choice. Several requests were made for more pictures and sounds. "A picture is worth a thousand words", one had answered. The ASCII version of text alone is impossible to accommodate pictures at this time, unless the pictures are printed as a hardcopy, and EVE is used as a "physical" encyclopedia. Another user asked for a simple, step by step, written instruction on how to get it running on a PC. The key point here was that SIMPLE instructions were requested. This is certainly an option for improvement. B.7 Proposed Improvements to EVE As a follow up to the evaluations taken on EVE, it might be beneficial to make additional improvements to EVE. Based on the questions asked in the survey, there are some realistic improvements that can still be made to EVE. These will take some effort from the individuals, but should not be so much that it is an overwhelming burden on any one person. An overview of EVE in general could be added. This is proposed to be the FIRST article of EVE (below the Table of Contents) that the reader would see as they (or if they) page down. This overview/abstract could contain: what EVE is, what virtual reality is, how to index. The Table of Contents could be modified such that the strings that appear in the TOC have exactly only one match to each of the respective articles. The users in the overview/abstract can be informed that if they do a string search on the strings in the Table of Contents, then the other matching string will put them right at the beginning of the article. It can be noted that this will be upper and lower case dependent as well. B.8 Conclusions Those that responded via email appeared to have longer answers that those who wrote out their replies. This may be coincidental, but perhaps people would rather type than write. EVE allows people to access information quickly. One of the main reasons for this is that EVE is electronic-based. Anyone with internet access can retrieve a copy of EVE and enjoy it many benefits. There were several variables that I was unable to keep constant when having users evaluate EVE. These were the type of systems they were using to evaluate EVE, the size of the screen that EVE was viewed on, and the B.9 Difficulties in Evaluating EVE The biggest difficulty in having users evaluate EVE was the fact that many of the users just did not have ample time to dedicate towards the evaluation. Because each article can take up to 15 minutes for some to read and comprehend, reading all of the 44 EVE articles would take 11 hours. This is just an example, but even three articles may take up to an hour to read, comprehend, and complete a survey. Many felt that they would like to continue reading but given their schedule, the time was not there. Because individuals participated on a purely voluntary basis, there were not as many surveys as I had hoped, but the quality was very high. C. - Bibliography [1] L. Sproull and S. Kiesler, "Connections, New Ways of Working in the Networked Organization", The MIT Press, Cambridge, Mass., London, England, 1991. [2] R. Fish, R. Kraut and B. Chalfone, "The VideoWindow system in Informal Communications", CSCW'90 Procedings of the Conference on Computer-Supported Cooperative Work, pp 1-12, Oct 1990 [3] J. Lee, "SIBYL: A Tool for Managing Group Decision Rationale", CSCW'90 Procedings of the Conference on Computer-Supported Cooperative Work, pp 79-92, Oct 1990 [4] C. Neuwirth, D. Kaufer, R. Chandhok and J. Morris, "Issues in the Design of Computer Support for Co-authoring and Commenting", CSCW'90 Procedings of the Conference on Computer-Supported Cooperative Work, pp 183-196, Oct 1990 [5] H. Ishii and M. Kobayashi, "Integration of Inter-personal Space and Shared Workspace: ClearBoard Design and Experiments", CSCW'92 Procedings of the Conference on Computer-Supported Cooperative Work, pp 33-42, Oct 1992 [6] B. Schniederman, "User Interface Design and Evaluation for an Electronic Encyclopedia",Cognitive Engineering in the Design of Human-Computer Interaction and Expert Systems,"Elsevier Science Publishers B.V., Amsterdam, 1987 - Printed in The Netherlands. ------------------------------------------------------------------------ Appendix A - EVE Production Survey and Results ---------------------------------------------------------- Survey Introduction: What follows are the survey questions that were sent out and the results summarized. Some similar answers have been combined, but this was a very time consuming effort and could not be completed. In a couple of answers, people asked not to be quoted. Where possible, the information is provided but in a way not traceable to the originator and information about specific partners removed. The following was explanatory text at the top of the survey. Everyone who received the survey had worked on EVE, so the introduction could be brief: Evaluating the production of EVE For many people, this was their first attempt at electronic collaboration. The class had no formal instruction on techniques or tools to facilitate this. We are trying to gather your opinions and also any techniques that you developed to accomplish your work. ---------------------------------------------------------- Question 1: If you worked with a student not at your same location, what problems were presented by working with remote students? A) I worked with two different students, in two different locations. In one case there were no problems working with a remote student. The other case, was very difficult... problems included: 1) no response in many cases to notes regarding logistics or draft edits. 2)when they did respond long response time between notes on planning and draft reviews and they didn't ever seem to have the latest version. 3) They submitted a different version (with only their portion of our article to the section coordinator. I didn't realize this happened until it was published back to me already graded by the instructor. B) Communication was restricted to e-mail, so it took a long time to make decisions. C) I did not have any problems writing the eve article together with my remote co-author. I could imagine that lack of motivation of one party (or both) can hinder the progress of the writing, since one can choose to ignore e-mail messages, and respond more slowly than if one had to meet with the person face-to-face. D) The writing was fairly easy except when we disagreed on emphasis, and that required a lot of back-and-forth email discussion, and even one telephone call. E) Quality control and coordination of each others tasks. F) Initial Communication of ideas became a problem. Since the telephone or email was used, difficulties arised when attempting to convey ideas only with words. Many times I wanted to grab a peice of paper and draw something out, something not easily done across the internet with inexperienced users. G) The delay in getting a response to a question, or in replying to someone else's question. H) Response time to email seemed to be slow, not sure if receiver got the message and ignored it, or didn't get it. I) We were in different time zones, so sometimes I would be working on it in the morning and needed to talk with my partner, who was probably not even awake yet because of the time difference. J) Two answered no problems presented by working with remote students. K) Several answers to this question were "N/A" because of either parnters dropping the course or getting signed up for a topic by themselves. L) The following are two administrative suggestions left in as feedback in case this project is repeated or a similar project attempted: M) I only worked with other students for coordination of sections, not on individual articles. My biggest complaint with this part of the collaborative effort was a lack of specific goals or a firm timetable of events. Jobs tend to seek their own priority level, and if the person coordinating the EVE effort didn't explicitly set a deadline for submissions, then EVE dropped to the bottom of the priority list. N) The REAL problem was co-ordinating the section. The remote student just didn't check email frequently, and I was left doing a lot of the work, but feeling obligated to keep the other person informed at every step so they wouldn't think I was trying to 'control.' ---------------------------------------------------------- Question 2: Was getting your group organized and deciding on the topic difficult? What made it easy, or could have been done to make it easier? A) N/A, since I worked alone. B) I'm having more trouble nailing down this JOVE project than I did with EVE. [Jove is the classes next project.] C) I chose topics with which I had some familiarity, which made topic selection fairly easy. However, there was some confusion about the number of topics that each student would take, and this made the effort to finalize the topic list somewhat more difficult. D) This was easy. In both cases my partner and myself signed up for the topic. I created an outline in both cases and recommended assigned sections. After 1-2 iterations on the draft outline we were organized. E) Getting the EVE group organized was easy - I flocked together with folks that had similar interests - picked up the topic first and then decided to co-ordinate with its co-author. F) No. The two groups I worked in got organized easily due to shared interest in the topics. G) Deciding on the topic was easy -- Mike and Claudio made the list and it was a matter of selection. The delayed broadcast for the remote students was a problem in a couple of instances because things had been decided between two of us, then we had a third person making contributions late. Working together would have been easier without the delay. H) NA I) It was easy J) For EVE, we just signed up for topics. The popularity of the topic determined the group and its organization. Since I didn't know anyone in the class, local or remote, organizing was somewhat slow, because I had to first establish some sort of connection with the person, i.e. figuring out schedules, how to collaborate, etc. K) Yes. One query went unanswered, so time was lost. For the other topic, the other person responded, so we had a group. L) No. Just signed up. Then the fun started. M) As mentioned, I wrote my article alone, so I didn't have there problems there. But when we were deciding the final EVE formats, we first threw some ideas around in E-mail, and then met personally to make the decisions. This method was quite easy as we had pretty good image of the possibilities before meeting. I think one or two new alternatives came up in the meeting, but as we had prepared using e-mail, the decision was fairly easy. N) I easily chose the topics I was experienced in or wanted to learn about. I only had a partner for one topic, and we split our topic (Force/Tactile Feedback) and did our own part, either force or tactile. ---------------------------------------------------------- Question 3: Was coming up with an original outline and starting the collaborative work of the body of the project difficult? A) Doing lots of initial research and reading helped greatly in writing an original outline. Before that, I have no idea what the issues are. B) No. Since I worked alone on both of my articles, I could organize them exactly as I saw fit. C) I did not perceive the collaborative part as difficult or hindering in any way. A couple of e-mail exchanges were enough to divide the work among ourselves. D) There was nothing difficult about the task -- in fact, it was a lot of fun! E) The EVE outline was already established. The subtopics within our topic was difficult when working with someone. When my partner and I broke our subtopic down into a very well (IMHO) structured topics, we found that we could work on our parts quite independently and yet have it flow. The problem was that writing on our sub-parts was just as long as a full article, so when we merged our parts, it was twice as long as Ben wanted. F) A little. Locating information was difficult. G) Not really, actually, the e-mail review session we had going between us really made it easier (since I work full time and didn't want the schedule crunch on face to face.) H) As you surely remember, coming up with EVE outline was kind of group task, but we needed one person to maintain the current table of contents of the offered articles. It was fairly complex as I recall. I) Again we each had our own subtopic, therefore comming up with an outline was an individual decision. J) Six answered No to this question. ---------------------------------------------------------- Question 4: What methods did you find to integrate the sections of your work? A) Two following methods were mentioned repeatedly and covered most cases: Method 1) We divided the project into sections. We both drafted our sections and merged them at the first edit. From then on we worked by passing the merged draft back and forth and highlighting changes with a >. Method 2) One partner wrote a first draft. It was then handed back and forth for updates or comments until both parties were satisfied. B) [One group had been using one of the above and had a person join their group late in the process.] I think it worked good until a third person joined our group. At that point, I felt like much of my hard work was practically deleted. C) In the unsuccessful collaboration, my partner conveniently omitted my portion of the article and therefore they sections were not merged. D) Leadership and good old delegation of tasks, with deadlines and reviewers for each piece. E) Sat down and hammered out the details on a piece of paper. Integrated vie-mail. F) Three answered "N/A" to this question. ---------------------------------------------------------- Question 5: What methods did you find to incorporate others comments? A) Huh? What methods can there be? If it's a useful comment, make the change. If not, ignore it. B) I sent them a 'thank-you' and modified my text as appropriate. C) I received comments after I solicited them for others in my section. I just incorporated those that I thought were appropriate. D) We highlighted comments and changes with a > and when we went through the draft edit we would either respond to the comments or delete them if the issue could be resolved. E) Just integrated the comments into the text - not aware that I used any special methods. F) Circulating the document among co-authors, and letting them add their comments worked fine for our group. It is of course important that only one person modifies the document at a given time. G) Usually, I read them carefully, and if I agreed with them, I just rewrote the piece. If I didn't agree, then I tried to persuade otherwise. If that didn't work, I either compromised (sometimes giving in, sometimes doing a modification of their suggestion) or I ignored it. (I tried not to ignore, though, because I think it's important to be democratic in a collaboration). H) Rewrote it incorporating the comments, either exactly what was suggested or something similar. I) An editing session before turning in drafts. J) Could suggest updates, and could send updated fragments. K) I kept a list of issues and comments to become an working report, and tracked the answers as we went along the timeline. L) I only got few requests for very small changes (mainly typos), so I just made those changes and my article was ready. M) We made direct changes to the documents. Upon receiving an edit. I would first read the general comments then the edited section. The general comments lead me to the changed sections. If I had no problem with the changes fine. If there was a problem I would replace it with the original with a reason for the change and send it back to my partner. ---------------------------------------------------------- Question 6: Did you find yourself compromising more or less than you would have working face-to-face? A) N/A. B) no C) No, because I didn't necessarily have to give a reason for rejecting a suggested change, I didn't have to compromise at all by making changes I didn't really want. D) The same. E) Less compromise via e-mail; more in face-to-face discussions. That's because it takes longer to communicate opinions by e-mail, so neither party has the patience to bully the other. F) I can't say for sure. G) Sometimes I think I did compromise more. It was just so HARD to keep going over the same point over and over trying to get someone to understand the point. However, I rarely give in when I'm convinced of something, so usually if I compromised it was because I didn't feel all that strongly and it seemed easy enough to do. (Don't quote me on this!) H) Yes, but mostly because of time constraints due to turn around time of email. Some comments I would have made in person, were not worth mentioning through the slow review process using email. I) Compromising more because it would take longer to resolve because of time zone thing and also because when we exchanged email, the turn around time is substantially longer than if you both were talking to each other at the same time. J) Maybe more, knowing how tight the deadlines were, and the delay in responses. K) Probably. L) About the same, I figure. M) I didn't need to compromise at any level of my tasks. N) I may have compromised less face-to-face, which would be bad. Be having the time to reflect on the email comments, I understood more clearly. Also there is a smaller tendency to compromise, rather than quickly defend the original section, when using email. ---------------------------------------------------------- Question 7: Did you experience any mis-communication because of no face- to-face contact? A) Yes, because deadlines for submissions were not always clear. B) Yes, as mentioned before a couple of times we had a mix up and were not working with the latest draft. C) Yes. Had there been face-to-face communication, the final product would have been better. More a case of dearth of communication than any miscommunication. D) Miscommunication? I think failure to understand the tone of a message happened once or twice, but nothing that didn't get worked out with practice. E) Yes, my partner mistakenly thought I had criticized her first version of our overview. Had we been face to face, I would have explained my ambiguous comment that was taken as negative. F) Yes...well, there were more times when something had to be clarified. G) Yes, in understanding how some of us with less capability to submit and review certain file formats were to get our work, en toto, to the team leaders. The resulting confusion over what do you have? What do "you" have? was painful and slow. One interactive phone call would have done it much better. H) That didn't happen to me, as I was able to meet with people face-to- face at all levels where collaboration was needed. I) Some ideas may have been missunderstood at first. However this was remedied with time. ---------------------------------------------------------- Question 8: Did you experience any flaming, or people being overly cautious not to flame because of the lack of face-to-face communication? A) N/A. B) no, though responses are typically very polite. C) None to my knowledge. D) No, I should have flamed at my one partner, but I let the mishap go. I probably would have let the mishap go face to face also. E) No flaming, not aware of any overt caution either. F) I think people avoided flaming because they mostly knew each other, and because they knew the professor was listening in. G) How would I know if they were being overly cautious? I experienced no flaming -- either sending or receiving -- and I gave none. H) Only slightly. Frustrations became evident. I) Not really, knowing the time delays, this would've been counterproductive. Besides, working on EVE was not really controversial enough to reach that point. J) No, but I was pretty cautiuos. K) How could I tell???? I don't think so, but not knowing the personalities, I had little on which to judge the responses. Second, I rarely "surf the net" and usually trash mail that is not concise. L) No flaming. M) I don't feel any flaming occured. We may have been overly cautious, but if so I feel that this did not have a negative affect on the article. N) I felt very uncomfortable commenting on someone else's writing and not being able to get any feedback on how it was being accepted, so yes, I was very cautious to repeatedly tell them it was there decision, these are only suggestions, and other statements like that. I also let small things go that I would have suggested about had we been meeting face to face and things been going well. ---------------------------------------------------------- Question 9: How would you rate your satisfaction with the final result compared to if you had done it yourself? A) Wouldn't have known. B) I was grateful for others' perspectives and feedback on how to improve my articles - specifically, I added several definitions for clarity. C) Because I *did* do my articles alone, I think they were better organized, but maybe not as comprehensive as if I had worked with someone else. D) In the one case I was VERY dissatisfied. I would have rather done it myself. I didn't feel the satisfaction I think I would have if I did it myself. In the other case I am neutral. E) Would have been better written, if I had done it by myself, but would not have incorporated diverse ideas. A tradeoff between the two proverbs: "Too many cooks spoil..." and "Two heads are better than..." F) The result better. Although I might not agree exactly with how things are stated in the article, working together with somebody on the article brought in ideas that I wouldn't have thought of on my own. G) Did you have to ask this question? :-) I felt very good about the article I wrote with Karl. I wrote it, then he made editorial suggestions and added a few points I felt really strengthened the piece. I felt I could have done a better job on the other one on my own. H) The only section that I co-authored was different than I would have done myself, and at some stages better. However, the addition of the third person to this process made this section of less quality in my opinion than what I would have written just myself. I suspect my other articles could have been better if I had a partner and had time to really collaborate with them. I) On one EVE project, I think I would have done it better by myself. The other, I was satisfied with. J) I didn't think that the EVE topics were big enough of a task to require more than 1 person working on a topic. Actually, I did my first article by myself which was less stressful. The second article I collaborated with a remote student, and even though I thought we had gone beyond the requirements of ouarticle to give it real quality, I felt like we were penalized because we didn't realize how long it got when we merged the two. There wasn't really any constructive critism when graded, except that it was horrible because it was too detailed. Not that I'm bitter or anything :-). K) My section alone was probably too short. However, the material added by the other person seemed to miss the point of the topic. After a suggestion from Ben on my other topic (worked alone), it seemed better then the combined work. L) Given the time I put into it, and knowing that our on-site coordinators did a whole lot more, I consider it to be better qquality that if I had done what I could alone, in the same amount of time. M) I was positively surprised about the project as whole. There were some details I would've done differently (and better :-)) but nothing major. N) Collaboration helped improve the final result. Not having feedback on the article I did alone produced an inferior product. I recieved a better grade on my collaboration article than on one I wrote myself. ---------------------------------------------------------- Question 10: How would you rate your satisfaction with the final result compared to if you had done it in a small group meeting face- to-face? A) N/A. B) I would not have wanted to do such a small project as EVE with several people, face-to-face or not. C) Because I *did* do my articles alone, I think they were better organized, but maybe not as comprehensive as if I had worked with someone else. D) In the bad case, I am not sure what would have changed. I think I just go a partner that doesn't have the "group" collaboration concept! In the other case, I would have had more satisfaction working face to face. E) A hypothetical situation about which I would not hazard to speculate. F) Hard to say. Nowadays, people don't write articles in face-to-face meetings anymore. I think the way people write articles together is more e-mail based, with comments inserted in the documents. G) Given the amount of time we had to work on this task, I think a group meeting would have helped resolve some of the points, but I'm not sure it would have made much difference overall. It was nice to be able to work late at night when there is nothing else in the world to distract -- and that happened a lot because I discovered that one of the people I was collaborating with worked late, too, so I was more likely to get a timely response then. H) I can only guess that I would have preferred to meet with the people face to face. That doesn't mean I would have had the time however. I) I think it would have turned out the same. J) It would've been the same. I think we would've agreed to find some method to work pretty much independently in chunks. K) Closer interaction would have helped keep the goal clearer. L) Same quality, but it took longer for the overall product. I think, and have for years, that same time same place conversations are better for converging or narrowing, and that currently e-mail is only an inprovement in divergent tasks or information sharing where time is not a major factor. Also, e-mail assumes some level of psychological sophistication on the both the sender and receiver, thus I probably would look for my assumption above to apply only to the "collegiate" atmosphere" and not the population in general. If it helps any, I think it is a factor related to patience (the trait). M) The full EVE wasn't produced by a small group :-), and I guess face-to- face meetings wouldn't have helped. N) If the project was performed by a small face to face group, I feel a better final result would occur. However the final result would occur a lot later. The collaboration with email created a environment which sped up communications over the total time of the project. ---------------------------------------------------------- Question 11: Did the project take longer, shorter, or about the amount of time that you had anticipated? A) It took shorter than I expected. B) about what I thought once I stopped procrastinating 8) C) Maybe a little less time than I expected. D) shorter E) took approx. the anticipated time. F) Shorter. One tends to get stuck sometimes working on one's own. Though people don't normally factor this into their predictions. G) I guess the project took a shorter amount of time than I anticipated, although it took hours and hours to write the one I thought was better. Writing takes a lot of time for me, period! H) I anticipated that it would take a long time, and it DID! I) It took longer. J) About as much time as I anticipated. K) Longer. I we had quit with final drafts and let the coordinator meak the end dicisions, I could have been faster (but not necessarily "fair"... ah, the trade-offs of work v. education. Since someone had to get a grade, the model for interaction was changed!) L) Putting the final version together required more work than I anticipated, but my own article was about what I thought it'd be. M) I would not have thought we would of produced such a product in the time of this semester. However, since EVE is not finished yet I am about right, EVE is taking as long as I anticipated. ---------------------------------------------------------- Question 12: We are interested in any other thoughts you may like to share about your work on the EVE project. A) It would be very helpful if each of us are assigned reading/grading of another group's work. Not only does it ensure high quality work, it will also keep us up to date on what other people are working on. B) What is the incentive to improve an article when the draft gets a letter grade? I corrected/edited the feedback I received and resubmitted, but I've heard that some groups received comments like "rewrite it". Unless there was a chance to improve the grade, I don't think I'd put more effort into such an article. C) After seeing the finished product on Mosaic, those articles that don't have pictures or sound (like one of my own) seem kind off disappointing. The multimedia nature of Mosaic really begs for some glitz. D) I tried to break out my two experiences. One was extremely successful and satisfying. The other a disaster. I think that group collaboration depends on the participants more than the mode of communication (face-to- face or remote). I found out my partner that did not incorporate my sections did the same thing to his other partner... it was just his way. E) I felt I could have done more. I also feel EVE on Mosaic is a terrific, ground-breaking effort. F) I think the project is interesting and worthwhile. I appreciated the instructor 'reminding' us from time to time about the goals and objectives -- I think this is important. As we started to work, it made him more of an impartial moderator -- if I reminded someone about what we were supposed to be doing, I felt as though I were 'whining' or that someone would perceive me as 'bossy.' I always threw out my ideas, often strongly expressed, with a comment like 'your comments, criticisms, and suggestions are welcome' to invite exchange on a deeper level. I really wanted our sections to be "good" and not just "completed." I think people have to care about what they're doing and be willing to discuss their feelings for this to work. G) Being the first time I participated in CSCW, I learned much. However, my email access is not easy to use. To print something takes much effort. I was forced to use word processors that I was not used to. My connection is slow and I have no easy way to respond to someone's mail message. I have no forwarding or other email features which make email easier to use. H) Overall, it was a major protion of what I learned in the class, and I think it should remain a part of any future class where remote learning includes understanding the distributed decision making process or distributed teams. I) I have never been associated with such a large collaborative work. EVE is a huge endeavour which I was a part of. Many members of this group I don't even know, however my articles, picture, and work on EVE is now "linked" with theirs. ------------------------------------------------------------------------ Appendix B - Actual Survey Responses from Evaluating EVE In this section, the survey respondents answers can be corellated via the letter next to the response. ie. All letters of A) are from a single respondent. Thus, you can match up responses to each sumbitted survey. Question 1: What are your overall impressions of EVE? (Electronic) A) The document is well written and provides a good overview of VR. B) I am very impressed with the content and the continuity of EVE. C) Overall, I think EVE is a very good product, and it should be useful for its intended audience. I do have a few reservations that I will list below. My sharpest criticism is for the lack of a good overview of the VR field. There is no good answer to the simple question "What is virtual reality?" Each article has lots of good detail, but the general overview of VR is missing. D) I thought it was well-done, although there was certainly more to be done in the area of linking the articles together... E) It's very nice and informative, but I'd love to have more embedded images in there. F) I have yet to see the entire thing since my computing capabilites are limited. G) It was very informative, well written, and well organized. H) A lot of interesting info. Easily accessible. I) To long, get to the point. J) There is a lot of good information. I would recommend an introduction section for people who are not familiar with virtual reality. Also an introduction will help to give the scope of this document. K) I found EVE to be both educational and entertaning. The thought of VR is as stimulating as your imanagination allows it to be. I was not aware of the advances that have been made in VR or the curent uses of VR. L) In a word, Impressive. This document covers all of the aspects of VR. M) Very good collection of articles. Informative overview of virtual reality and its various aspects. (Handwritten) A) Good Info. Liked the Voice Synthesis Article B) Very comprehensive in dealing with various areas. C) Interesting. I would continue to read articles if given time. ------------------------------------------------------------------------ Question 2: What is your impression of the consistency between the articles? (Electronic) A) Some consistency problems definitely exist. I think it reflects the style differences of the writers. It may have been good to have a master proofreader go over things to increase consistency. B) The articles suprised me on the consistency they had to each other, since so many people worke on it. The work of the group leader must have been substantial to keep all of the articles "looking" the same. C) I think there is good consistency between articles in terms of general writing style and technical level. However, some articles had good overviews leading into the detail (I'm big on overviews!) and others would just dive right in to the heavy stuff. Also, because some articles had good graphics and sound links, the ones that didn't really stood out as being drab. D) Somewhat varied, although, at least the formatting was pretty similar in the hypertext version. E) There is quite a variation between the articles. Some are fairly high-level easy to read stuff (which I think was the goal), but some articles go to unnecessary detail especially in some parts that have very little to do with actual VR. F) It varies widely, as I would expect given the timeframe and the background of the authors G) Consistency was very good. Information that over-lapped between articles was just enough to allow the articles to stand alone outside of eve. H) Pretty good, considering the number of different authors. I) Good,however I like fact stated first. J) All the articles blend very well. K) I found all the articles I read to be a little redondent in places, however all the articles were very inlightining (spelling?). L) Very consistent. M) Not that good. Some discuss educational requirements, others do not. Some have bibliographies, others don't. Some are veryconversational in style and easy reading, others are just lists, with very broken text and are more difficult to follow. (Handwritten) A) Good - Could provide more explanation or what is being tried in today's technology. B) Format changes between the articles and there is apparent difference in English skills. C) Given the diversity of the authors, there seems to be an average level of consistency. ------------------------------------------------------------------------ Question 3: Many articles were produced by pairs of authors using email. What is your impression of the consistency within articles? (Electronic) A) This level of consistency seems very good. B) I feel the consistancy is good. That probably is because email is very useful for collaborating. C) I was not able to detect any abrupt changes in writing style or content. D) In general it was not apparent they were cooperative work - which I consider to be a compliment. E) Individual articles are mostly very well done. F) It's okay given the intended purpose. G) I did not notice and continuity problems within articles. H) Real smooth - sounds like one author I) Good with some overlap. J) Individual articles also blend very well. K) All the articles I read that were written by pairs of aurthors were well written. The consistency of the articles was very thorough. L) Very consistent. M) Multiple author articles seem consisant within themselves, but not necessarilly consisant beteen different ones. (Handwritten) A) Good B) N/A C) Overall there was a high level of consistency on articles written through email. ------------------------------------------------------------------------ Question 4: How do you rate the readability of the articles? (Electronic) A) Most are very readable, excellent. B) Most of the articles are fine. However a few do get a little wordy. It is unlikely that EVE will be assigned reading, therefore it should be enjoyable and easy to read. C) I would say that there weren't any articles that were not readable. D) Quite good. E) As mentioned above, some (most) are easy to read, but some articles go in too much detail. F) Good to excellent, except one that's terrible. G) All of the articles were quite readable and very interesting. I found myself reading pieces from many categories that I did not initially find to be of interest. H) Very readable and understandable. I) Good,short but as in question 1 above, I like fact first. J) Good K) The articles were very easy to read, with just enougth technical terms to make them informative. L) Requires an 11th grade reading level. M) As mentioned above, some were easier, some harder for the above cited reasons. (Handwritten) A) OK, could use some extraction of non-useful sentences and down to the point B) Good, no great though C) They are very readable ------------------------------------------------------------------------ Question 5: How do you rate the technical/information content of the articles? (Electronic) A) Some articles do not seem to address the entire scope of the topic rather some isolated pieces. Other than a bit of this, the information seems adeqaute for an undergrad comp sci student. B) THe technical information is very good, almost to good. So much technical info is in some articles that it makes it difficult to read C) About half of the articles had the right technical information content but had too few examples. The other half tended to go into to much depth. D) Decent - could be a little deeper in some cases. E) Information content is in most cases ok. Again, in some cases it's too high. But I'm repeating myself. F) high-school advanced science students could use it, otherwise, collegiate early developmental. G) I thought the articles had been "dumbed" down enough for users without a VR background. The glossaries were also helpful. H) I) Good. It is said that for every math equation used in your article the writer will loose about 10% of their readers. I guess that is the difference between peasure reading and reading to learn. J) Good K) All of the technical terms were well explaned. L) The information is very informative. Some of the articles could be more in-depth, but they cover most topics adequately. M) For the most part, technical content was very good. Subjects were explained in enough detail to understand them without getting lost in jargon. Articles with references have tha added advantage of further reading capabilities if the reader desires. (Handwritten) A) Good and informative B) Good C) Fair ------------------------------------------------------------------------ Question 6: EVE was targetted for undergraduate CS students. Do you think that the writing was appropriate for this level? (Electronic) A) yes B) Yes, if this is assigned reading. I'm not sure that all articles would be read by a passer by. The reading can be understood by undergrads, but a few articles my not spark interest due to their wordiness. I felt that the interest was what was of importance for EVE. C) Again, about half the articles were too technical for the casual undergraduate CS student. D) Probably - I still think they could have been a bit more meaty. E) Mostly yes. I won't repeat anymore. F) Matbe a little weak, but that's expected due the limited overall CS background og some students. G) yes. H) There may have been terms / words that make it less understandable to someone who is not pretty familiar with the technology than it could be. I) Yes J) Yes K) Yes and No. If they were to inform th ereader in a enjoyable matter then yes. However if they were designed to instuct the reader then NO, all the articles were to simply written. L) See #4. M) Definately. Even those without knowledge of the subject would gain a good understanding of it. (Handwritten) A) Yes B) Yes C) Yes ------------------------------------------------------------------------ Question 7: EVE was partly intended to provide information to undergraduate students on the classes to pursue in order to equip the reader with abilities relating to Virtual Reality. Do you think the recommendations were appropriate? Too many? Too Few? (Electronic) A) yes, although this is sometimes too vague a subject. I think it would have helped to give, in addition to general recommendations, a specific example set of courses at umd. B) There were too few recommendations. However this is due because there are few classes around that would help a student prepare for VR. Only parts of a few classes would provide useful information. C) I think there were too few specific course recommendations. Suggestions like "study lots of math and science" aren't going to help much. D) Mostly pretty good. E) Some of the recommended classes were quite far off from CS which I feel is the most important skill. It would be ok if they were introductory courses, but there were some suggestions of pretty "deep" courses in related areas, which probably should've been left out. Also some articles recommended almost all possible CS courses. VR requires lots of skills, but the authors could've at least ranked the courses somehow. Overall, the articles that had recommendations often had too many, and then quite a few articles didn't have any recommendations at all. Though I don't think e.g. the physiological articles needed any recommendations - that would go too far from VR. There is only a limited number of courses you can take as undergrad. F) Not really, since I think each aera has more to offer as a subject than we were able to convey, mostly due to time constraints. Too many? Number of topics ok, the scope too limited. We went wide, and maybe in some cases should have gone deep. Tell tham something they can't find out easily somewhere else. Too Few? Ok for the depth, as stated above. G) The recommendations were good but did not describe necessary class material, i.e. optics, controls, etc. H) Good recommendations. I) I would like to see more introduction in the articles. J) Appropriate K) Once again Yes and No. Most of the articles I read did give ample information concerning classes of other related sorues to follow up on more VR info. There were a few articles I read that were a little shy in this area. L) Too few. M) When this was listed, most of the time it was OK. Several articles had no such listings, and some were very generic in the recommendations. (Handwritten) A) Don't know? B) N/A C) Far too few explicit recommendations ------------------------------------------------------------------------ Question 8: Any other strong points to EVE? (Electronic) A) Not that I can think of. I think its a good piece of work. B) EVE is definetly a first step to looking into the area of VR. The articles give a good overview in all the aspects of VR. I am proud to be apart of this class's work in creating EVE. C) Apart from the lack of an overview, I think the logical organization of the sections and subsections was good. D) There's a lot of information there! Coverage is good for a one-month project. E) The Mosaic version is accessible from anywhere on the planet. And the ASCII version is also available. F) It certainly gives the interneters something to recognize, that it's (VR) becoming a major interest of the academic/research UNDERGRADUATE community Few schools offer a comprehensive program in VR/TP G) H) Very interesting. I) Good information J) Good reference for people interested in more information on the subject K) Well written!! Highly enjoyable!! Very informative!! L) M) Good intro/reference on the subject. (Handwritten) A) A lot of info to digest - may take a second or third reading for total understanding B) Very interesting. Kept my attention. C) None ------------------------------------------------------------------------ Question 9: Any other weaknesses that stand out? (Electronic) A) I think it needs an abstract and some method of indexing to the article of choice. I also feel it needs a section dedicated to subjects worthy of mention but not discussed in EVE. Certainly, we did not cover all of virtual reality in this thing. B) The essence of VR is the multimedia. More pictures, sounds, movies would help show what aspects of VR can do today on your computer screen. The text is written well, however more "sizzle" is needed to create a proper VR document. C) When taken out of context, as they are in the Definitions section, some of the definitions are hard to understand. As an example, the definition of annular ligament is "attaches the stapes to the oval window." Now I have to look up stapes and oval window. A little more explanation to say that these are structures in the middle ear would have been helpful. D) Nope. E) The coverage ended up being pretty uneven. Some areas are divided into many articles and deep hierarchy (leading to problems with too detailed information) and some other areas are left with very shallow coverage. But this probably results from the way it was put together. Everybody was picking a topic of interest, instead of first creating the table of contents and article headings and then selecting topics from there in such a way that all tpics would've been covered. F) Simple, step by step, easy to use, written for the layman instrunctions on how to get it running on YOUR personal computer. I know the audience is CS types, but why can't it be written in plain english, no jargon should be used in instructions...too much ambiguity. G) H) I) None J) I did not see any illustations, since I looked at an ASCII version of the document. A picture is worth a thousand words. K) None L) M) It would benefit from a more consistant style. If a single person was proofing and suggesting improvements, there would not be the style breaks as in the current document, and readability would be much improved. (Handwritten) A) None A) Diagrams and double spacing may make readability easier. A) None ------------------------------------------------------------------------ Appendix C - Credits Mike Burton performed the following: Produced the EVE production survey. Produced a draft of the EVE evaluation survey and forwarded it to Gary Withrow who updated and distributed it. Produced a draft of section A.1 and forwarded it for Gary's description of section B and updates. Collected and organized the results of the EVE production survey. (This was MUCH more time intensive than anticipated.) Produced sections A.2 through A.4 and forwarded them to Gary for comment. Produced a draft Bibiliography with enties 2 through 5 and forwarded to Gary Withrow for his references. (Both of us used reference 1) Searched the Boeing Technical Library to find reference materials, performed research in the 1990 and 1992 CSCW procedings. (We don't all have the advantage of a college library. I am lucky the Boeing library is as good as it is, some remote students may have had an even harder time with this part of the assignment. HCIBIB gave me 15 references, the only ones I could find in Boeings holdings were the two ACM CSCW conferences. Luckily, they were full of great papers.) Reviewed Section B and provided comments. --------------------------------------------------------- Gary Withrow performed the following: Produced a draft of the outline page and forwarded to Mike Burton for inclusion of the first survey. Made revisions for a final copy of the EVE evaluation survey for distribution. Originally combined all individual text article files to have one large file, but after learning that this was not the latest text version, I unzipped the PC version of EVE in ASCII. Created the directory /pub/eve on UMD5.UMD.EDU placing in the latest ASCII version of EVE, the survey file, and the credits file. (FTP back and forth to UMD of large files is not as good as moving files on a remote machine with FTP - but can you move files with FTP?) Posted the EVE Evaluation survey onto the internet newsgroups: alt.cyberspace, sci.virtual-worlds, alt.uu.virtual-worlds.misc, sci.wortual-worlds.apps. This was the request of all those "out there" to use and evaluate EVE. Created a bounded hardcopy version of EVE in ASCII. This included a separate folder cover page. (Small font to accomodate the large file) Distributed the EVE Evaluation to the VRTP class. Recruited relevant evaluators including work associates, friends, CS undergraduate students, and classmates to evaluate EVE. This was the most difficult part since many were "hard pressed for time". References pertinent to the EVE Evaluation survey are 1 and 6. Revised section A.1 (abstract) and forwarded to Mike for comments. Produced sections B. through B.9 and Appendix B. and sent to Mike for comment. Reviewed Section A and provided comments.