GROUP COLLABORATION IN THE VIRTUAL CLASSROOM: an Evaluation of Collaborative Learning in the Virtual Classroom of CMSC 828S and the Technology that Supports It Deborah Barreau College of Library and Information Services University of Maryland College Park, Maryland barreau@wam.umd.edu Cheryl Eslinger Virginia Polytechnic Institute Science Applications International Corporation eslinger@cseic.saic.com Kim McGoff rmcgoff@owgvm3.vnet.ibm.com Cynthia Tonnesen School of Engineering and Applied Science The George Washington University Washington, D. C. tonnesen@enews.nrl.navy.mil ------------------------------------------------------------------ ACKNOWLEDGMENTS Sincere thanks are given to Ben Shneiderman for his innovative teaching and his valuable suggestions for this project. A special thanks to Theo Stone for his generosity in lending us the MCI video phones; to Toni Emerson for recommending electronic sources on the Internet, and to Starr Roxanne Hiltz for assistance in locating the Shavelson article. Appreciation is extended to our readers, Glen Lee, Dan Searles, Marko Teittinen, and Gary Withrow, for their helpful perspective on our writing. Last but not least, thanks to the students of "Virtual Reality, Telepresence, and Beyond" for their time and thoughtful comments on our long survey. Their participation was vital to this project. ------------------------------------------------------------------ TABLE OF CONTENTS 1. Introduction 1.1 Overview of Distance Learning 1.2 Characteristics of Distance Learning and Virtual Classrooms 1.2.1 Features of Distance Education 1.2.2 Virtual Classrooms 1.3 Studies and Findings 1.4 Implementation of the Virtual Classroom in "Virtual Reality, Telepresence, and Beyond" 1.4.1 Goals of the Course 1.4.2 Issues and Implementation 2. The Research Problem and Hypotheses 2.1 Group Collaboration 2.1.1 Ease of Access 2.1.2 Sense of Engagement 2.1.3 Sharing of Information 2.1.4 Individual Gratification 2.2 Hypothesis 2.3 Technology Supporting Group Collaboration in our Virtual Classroom 2.3.1 Electronic Mail 2.3.2 Video Phone Case Study 3. Research Methods 3.1 Class Evaluation 3.2 Video Phone Case Study 4. Results 4.1 Survey Support for the Critical Factors of Group Collaboration 4.1.1 Ease of Access 4.1.2 Sense of Engagement Sociogram Support for Engagement 4.1.3 Sharing of Information 4.1.4 Sense of Gratification 4.2 Analysis of VRTP Listserv Mail Student Involvement and Participation (dropouts) 4.3 Results of Video Phone Test Trials 4.3.1 The State of Video Phone Technology 4.3.2 Video Phone Usage in the Virtual Classroom 4.3.3 Small Group Collaboration with Video Phones 4.3.4 Video Phone Conclusions 5. Conclusions Critical Success Factors Limitations of the Study Technology, Technique and Distance Learning ------------------------------------------------------------------ ABSTRACT The university classroom of the future may bear little resemblance to the instructor-centered environments common today. Distance education has been part of university curricula in the form of correspondence and telecourses in the past, but the maturity and affordability of such technologies as interactive compressed video and electronic mail offer the potential for highly interactive, student-centered learning environments which are as satisfying and engaging as "being there." This environment is established, in the view of one professor, through successful collaboration among students who are actively engaged in the construction of "something meaningful and substantial."* This paper is an evaluation of computer science course, "Virtual Reality, Telepresence and Beyond," offered to graduate students, both on- and off-site at the University of Maryland in the fall of 1993. The evaluation addresses the question of what makes a collaboration successful, assembling data from student surveys, class observations, a student journal, and electronic mail to test hypotheses that remote students will not differ from on-campus students in their reporting of successful collaboration in the virtual classroom. Although problems were reported in sharing information and in the feelings of isolation of some remote students, the majority of the remote and on-campus students reported high levels of satisfaction in the virtual classroom. *Ben Shneiderman, Professor, Department of Computer Science, University of Maryland. ------------------------------------------------------------------ 1. Introduction 1.1 Overview of Distance Learning The concept of the virtual classroom described here is an extension of distance learning, or distance education, which occurs when there is a physical separation of instructor and student. Traditionally, distance education courses offer students who are unable to attend conventional classes the opportunity of an education, offer educational institutions an opportunity to expand enrollments, and offer businesses and other organizations alternatives for staff development and training. Some distance education programs, such as the National Technological University (NTU) have been highly successful, yet many question whether the quality of the education is comparable to traditional programs, and many of the students enrolled in these classes still indicate a preference for attending courses on campus when opportunity permits (Barron, 1987). Recent developments in technology, accompanied by cost effective means of delivering instruction electronically, have stimulated interest in distance education. It is now feasible to offer remote students full, interactive participation in a class that would previously have been restricted to students who were attending locally. These technologies include such alternatives as two-way interactive, compressed video and electronic mail. One of the advantages of these technologies is a greater ability to support collaboration among students as well as between student and professor (Bricken, 1990; Shneiderman, 1993). The potential for the technology is to create virtual classrooms and virtual laboratories which can exist beyond the traditional boundaries of a particular university or location to a broad universe of students, faculty, and resources (Silverman et al, 1993). Studies have shown that successful distance learning occurs when the tasks and activities are appropriate to the technology; are consistent with the instructor's philosophy and style of teaching; are convenient, accessible and relevant to the students; provide maximum interaction; and are well-organized and well-presented (Barron, 1987; Bruce, Peyton, & Batson, 1993; Bulkeley et al, 1992; Carroll, 1993; Faibisoff & Willis, 1987; Hiltz, 1993; Hiltz, 1986; Hiltz & Meinke, 1989; Osterndorf et al, 1992). The following section is a discussion of the issues and features of distance learning. It is followed by a description of the specific implementation of these features in the course, "Virtual Reality, Telepresence and Beyond," and an evaluation of the course based upon student responses to a survey, observations of communications in class, and electronic mail shared on the class electronic "listserv." 1.2 Characteristics of Distance Learning and Virtual Classrooms 1.2.1 Features of Distance Education Distance education was developed to reach students who find it difficult to attend traditional classes due to distance, personal difficulties and responsibilities, or time constraints. Students who enroll in distance education courses tend to be aged 20-40, employed at least part-time, and sometimes unable to meet the entrance requirements to regular programs (Faibisoff & Willis, 1987). Methods of delivering distance education include student travel to campus, instructor travel to student locations, videotaped lectures, live video via satellite or local microwave with two-way audio by telephone, and two-way interactive compressed video (Bulkeley et al, 1992). However, it is misleading to imply that these methods are neatly and uniformly implemented forms of delivery as research reveals that each implementation is unique (Hiltz, 1993; Bruce, Peyton, & Batson, 1993). Distance education offers students convenience and flexibility, allowing them to focus more on the learning experience by avoiding lost time in travel, and it provides access to human resources beyond the local site. The disadvantages associated with distance learning include a sense of confusion or feeling of "disorientation" with the class which may be a function of adjusting to the medium or experiencing delayed broadcast, occasional technical difficulties, lack of adequate resources at the student site, and a sense of isolation from the class. Distance education can also pose problems in coordinating assignments, examinations, and providing for adequate student feedback. As the state-of-the-art for delivering courses remotely becomes affordable and available, and as individuals, corporations, and universities become more aware of its advantages, there is likely to be greater interest in distance education. Much of this is due to the proliferation of computers and electronic networks which offer rapid access to a global community of information resources. 1.2.2 Virtual Classrooms The use of live broadcast, interactive video, and electronic mail makes a "virtual classroom" without walls in a university without a campus conceivable. Electronic networking provides access to thousands of resources, human as well as data, all over the world. Electronic mail has made it possible to communicate with individuals, including technical experts, politicians, and friends within minutes. This technology has been used to support distance learning to create greater student involvement and participation (D'Souza, 1992; Hiltz, 1986; McClelland, 1993; Bruce, Peyton, & Batson, 1993), providing the opportunity for more personalized one-on-one communication between student and teacher and between students, facilitating the exchange of ideas, and supporting regular review of project status. While the technology is available to make this possible, successful implementation of the virtual classroom is dependent upon the factors already mentioned above, tasks and activities which are: 1) appropriate to the technology; 2) consistent with the instructor's philosophy and style of teaching; 3) convenient, accessible and relevant to students; 4) highly interactive; and 5) well-organized and well-presented. Research has offered additional insight into the ingredients of successful distance education. 1.3 Studies and Findings Central to an evaluation of any distance education program or course is an understanding of the variables which make a successful experience and methods for measuring those variables. Studies have shown that while the factors mentioned above are most important to successful presentation of material in virtual classrooms, the factors which influence student performance and subjective satisfaction vary, and include student attitudes and preparedness. Student level of participation has proved to be a consistent predictor of reported satisfaction in many studies (Hiltz, 1986; Hiltz, 1993; Hiltz & Meinke, 1989). One of the features which makes virtual classrooms different from traditional remote classrooms is that they are not bound by time or place. When such classes are accompanied by electronic communication, class discussions and projects can continue when appropriate. Such communication makes classes more fun for students and enhances the interactive and social components of distance learning. Studies show that the nature of communication is different in electronic environments (Chapanis, 1975; Bruce, Peyton, & Batson, 1993; Hiltz, 1986; Hiltz & Turoff, 1985). For example, in electronic environments, individuals have been shown to be less inhibited and less formal in their communication, engaging in behaviors such as flaming and writing loosely structured prose with misspellings and often with words running together. Some suggest this flaming is the result of the "depersonalization" of the environment, of the awkwardness and difficulty of communicating without nonverbal cues. Initially, students tend to experience some confusion in adjusting to new software or systems, sometimes reporting cognitive overload and lack of productivity, but studies show that after some period of adjustment, they tend to become proficient and learn to manage the system. Hiltz suggests that giving each student personal attention and time to adjust to a system before making complex assignments can help to minimize the difficulty experienced. The informality and flexibility of virtual environments poses unique problems as well as opportunities. Students frequently report that when they get busy with other things, they are more likely to stop participating in a televised course than to cut a traditional class, for example. It is not only the informality and impersonality of the medium which pose problems for distance learners, but how the technology is used. Successful use of the technology is a critical factor in student satisfaction, and studies have shown more positive response occurring when professors are comfortable in front of the camera, with a natural, relaxed style of presentation, shifting content or camera angle every twenty minutes or so, using clear visual aids of about 10 lines in length, and maximizing interaction (Bulkeley et al, 1992). Evaluating the virtual classroom poses many problems for researchers. For example, students enrolled in distance education are self-selected rather than random, indicating the population is somewhat unique from traditional environments and making it difficult to generalize from results. Additionally, each implementation is so unique that it becomes difficult to generalize the experience to other classes, even when content is the same. Typically, evaluation of distance learning has focused upon student performance on standardized tests, grades, or student satisfaction as reported in response to pre-course and post-course questionnaires, classroom observations, interviews, dropout rate, or willingness to enroll in additional distance courses (Hiltz, 1993; Shavelson et al, 1986). These studies are useful and important in considering the value of distance education to student, educational institution, and the organizations which employ the students. The following section describes a particular distance education course offered in the fall of 1993, and includes a description of the goals of the course, the goals of the evaluation, and a framework for the evaluation. 1.4 Implementation of the Virtual Classroom in "Virtual Reality, Telepresence, and Beyond" 1.4.1 Goals of the Course "Virtual Reality, Telepresence and Beyond" was offered for the first time at the University of Maryland in the fall of 1993. The course, CMSC 828S, was also offered as CS 790B from NTU. The course was offered to provide students with the opportunity to work remotely by way of broadcast lectures/discussions, electronic mail, and computer conferencing techniques to explore both the promise and the reality of VR applications. All activities, including individual assignments and team projects, were designed consistent with the professor's philosophy of engaging students in the active construction of something meaningful and useful (Shneiderman, 1993). Activities included class presentations, discussions, readings, two examinations (reduced to one due to time constraints), a group project to develop an on-line Encyclopedia of Virtual Environments (EVE) and a research project resulting in a paper which could be compiled and produced as a Journal of Virtual Environments (JOVE). Students also extended class discussions through electronic mail to individuals or subgroups and to a distributed mailing list or "listserv" which reached all participants. Students were invited to participate in defining the scope and content of these projects as well as to manage and coordinate project development. The professor stressed from the outset high expectations for the quality of these projects, sending both a challenge and a vote of confidence to the group. 1.4.2 Issues and Implementation The implementation and presentation of course information and class assignments were focused and effective, features which were identified above as beneficial to distance learning. Class activities included videos which are appropriate for conveying information in a broadcast environment. Both student presentations and group projects were consistent with the instructor's philosophy of teaching. The professor seemed comfortable with the technology, instructing the technical assistant to vary the camera angle as appropriate and shifting topics frequently to maintain interest. The technology introduced some constraints to the system, however, which require attention here. Possibly the greatest constraint was due to problems related with the broadcast. The professor did not discover until the class started that live broadcasts would reach only those remote students in the immediate viewing area (three students), while other remote students received video-taped broadcasts of the lecture two to seven days later. There were a total of twenty-five students enrolled in the course for credit, and twelve of these students were located at remote sites. This fact greatly impacted the ability for remote students to call in and participate in discussions. While two of the three students viewing the live broadcast took advantage of the opportunity to call in and participate, and one became an active participant via video phone, it was not possible for the others to do so. Students adjusted to the technological constraints, with on-campus students sending class notes to remote students to ensure timely delivery of class business and with students interacting by telephone, face-to-face meetings, FAX, postal service, and electronic mail to organize and develop class projects. Since a major goal of the course focuses on the collaborative efforts of students to produce worthwhile projects, the course will be evaluated in terms of how successful groups were at collaborating remotely and how successful the available technology was at supporting these collaborations. Student reports of success in collaboration and of satisfaction with the supporting technologies are also used as measures of the success in this virtual classroom. 2. The Research Problem and Hypotheses "Scientific collaboration provides a model of the way professionals in many fields construct intellectual products" (Kraut, Egido & Galegher, 1989). The study of group collaboration in the virtual classroom may help specify the technological needs of cooperating work groups, as well as the limits of this technology. In the virtual classroom students are collaborating in a space that doesn't actually exist and where social norms for group collaboration are not well established. Shifts in interaction patterns and group dynamics result from working in a virtual collaboration space. The fundamental objective of collaborative learning is the transfer of knowledge between members of the group. The traditional educational experience can be typically characterized by verbal dialog in a same time, same place; instructor to peer or group collaborative environment; using an established, well defined set of social norms. In the virtual classroom the traditional knowledge transfer experience is changed to an independent time; independent place; and computer-mediated group collaborative environment. The virtual classroom transforms traditional education into an experience that may be independent of time and location, and knowledge may be transferred in ways other than teacher lecturing to student. The virtual classroom can be considered a system that combines students and the professor with communication, computer, and video technologies in a multi-user education application for virtual group collaboration. Group collaboration tasks include, but are not limited to planning, problem solving, issue discussion, negotiation, conflict resolution, document preparation and information sharing. The premise of this research is to show that if the technology and elements that support successful collaboration are present, then cooperative efforts can be successful. 2.1 Group Collaboration Group collaboration occurs when a collection of people sharing ideas, information and similar objectives participate in a collective knowledge transfer experience. Previous group collaboration research has evaluated the effects of physical proximity, group size, and communication concurrency on various group processes using technologies such as teleconferencing, group decision support systems and electronic meeting systems (Jessup & Tansik, 1991; Kraut, Egido & Galegher, 1989; Dennis, George, Jessup, Nunamaker & Vogel, 1988; Valacich, Paranka, George & Nunamaker, 1993). Moreover, the research often focuses on single or narrow, albeit important task of the group collaboration while measuring variables such as efficiency, attentiveness, productivity, participation, relationship longevity and satisfaction (Horowitz, 1988; Dennis, George, Jessup, Nunamaker & Vogel, 1988; Kraut, Egido & Galegher, 1989). Additional research with computer-mediated communication systems (CMCS) has analyzed user satisfaction, system acceptance, amount of use, and benefits (Hiltz and Johnson, 1989 and 1990). Hiltz and Johnson suggest that, "future work on CMCS's should not assume that usage or subjective satisfaction alone are sufficient indication of a successful system implementation." Research results have varied depending on the task evaluated. Most researchers conclude the technologies, procedures and guidelines must be optimized for each specific collaborative task or project. This research highlights considerations for further study of group collaboration. Specifically, correlation of proximity (remote versus on-campus), group size (whole class and small project groups), participation, benefits, satisfaction and acceptance of the technologies will evaluate the success of our group collaboration experience. Based on the results of the previous research it is apparent that there are a number of critical factors that influence the success of each collaborative project. Related factors can be combined to create four factors critical to the success of any virtual collaboration. We believe that successful group collaboration learning in a virtual collaboration space requires: (1) ease of access, (2) sense of engagement, (3) sharing of information, and (4) individual gratification. These four factors encompass most of the variables and factors studied in previous group collaborative research that are applicable to the virtual collaborative learning experience. These four elements are the critical success factors (CSFs) we will evaluate to determine the success of collaborative learning in our virtual classroom. The CSFs are defined in the following paragraphs to provide a basis of understanding for our analysis and results. 2.1.1 Ease of Access In a previous study of the virtual classroom Hiltz concluded that a successful outcome (for the Virtual Classroom) requires adequate access to necessary equipment (such as computers and modems, for example), so that both students and instructors can easily access the facility. Our research evaluates the ease of access to the collaboration. The evaluation of ease of access includes measures of convenience, efficiency, student expertise, and analysis of technical difficulties with electronic mail, video broadcasts, and other supporting technologies, such as video phones. 2.1.2 Sense of Engagement According to social presence theorists, visually-oriented face-to-face communication is socially oriented and rich, supporting informal communication because it highlights participants interaction and interpersonal relations. (Fish, Kraut & Root, 1992). Written language in electronic form lacks two key features found in face-to-face dialog: (a) the necessity to respond to and coordinate with another person, on-line in real-time and (b) the use and integration of precise nonverbal as well as verbal elements (Bavelas, 1990, Bavelas & Chovil, 1992). Given the asynchronous, non-verbal nature of the electronic mail and video taped communication used in our virtual classroom, we anticipate the sense of engagement that is created will compensate for the lack of visual and verbal cues. Our evaluation of a sense of engagement includes an analysis of the presence of real or perceived sense of urgency in timely response, remote student attentiveness to virtual classroom discussion, students' sense of "being there" or involvement in the virtual classroom, and students' sense of relationship with peer students they have never met face-to-face. 2.1.3 Sharing of Information Studies conducted by Hagstrom, Kraut and Egido indicate, that informal contact that results from frequent opportunities for communication often leads to collaboration. Allen (1977) showed a striking logarithmic decline in communication frequency with distance between potential communicators. For example, in Allen's data, about 25% of the subjects whose offices were next door to each other (less than 5 meters apart) talked to each other about technical topics at least once a week; if their offices were 10 meters apart, this figure drops below 10%. After this sharp decline, the curve asymptotes at approximately 30 meters, so that subjects 30 meters apart and those several miles apart had approximately the same low probability to talking to each other at least once a week. Given the different time, different place nature of the virtual classroom, it is necessary to evaluate what implications differences in proximity (remote versus on-campus) have on student collaboration in the virtual classroom. Equally important is an understanding of the technologies and techniques the students employee to compensate for the asynchronous nature of the collaboration. Our analysis of sharing of information evaluates the amount of student participation, what communication medium students preferred, who students chose to partner with in collaboration (local or remote), and student subjective feelings about information sharing (constraints, conflicts, and successes). 2.1.4 Individual Gratification Studies show that subjects who participate more in group discussion and collaboration are more satisfied (Hiltz, 1986; Hiltz 1993). In addition, Kraut and Egido demonstrated that researchers who are situated near each other are likely to come to like each other more and in turn have a strong influence on the likelihood of collaboration. The individual gratification reported by the students may be correlated with other factors (ease of access, sense of engagement, and sharing of information) to provide verification and insight into the importance of a given critical factor in the success of the virtual classroom. This analysis of the correlation of gratification and other CSFs will reinforce the importance of a given CSF and provide insight into potential technologies, techniques and experiences that contributed to satisfaction in the collaborative learning environment. Our evaluation of individual gratification test students' perceptions of benefits, information overload, fairness and equality, quality of collaboration outcome, satisfaction with working relationships, and if they would choose to take another course in the virtual classroom. 2.2 Hypothesis Based on the definition of successful group collaboration, we expect remote students will not differ from on-campus students in their reporting of successful collaboration in the virtual classroom of CMSC 828S. The hypothesis is evaluated for both the general classroom collaboration (which includes regularly scheduled, broadcast lectures and discussion on electronic mail) and the specific small group project collaborations. Our evaluation extends to analyze the specific communication technologies, electronic mail and video phone, supporting group collaboration in our virtual classroom 2.3 Technology Supporting Group Collaboration in our Virtual Classroom Electronic mail, video phones, and other technologies are available to support collaborative endeavors. In measuring the success of collaborations for remote and on-campus students, we have analyzed how the technologies we used, support the four factors of collaboration. The better the technology supports these factors, the more probable the technologies will help overcome the limitations of distance learning. The following sections describe how the technologies we used did or did not support the CSFs of virtual collaborative learning. 2.3.1 Electronic Mail The proliferation of personal computers and the growing accessibility of internet and other widespread computer communication services has led to the possibility of on-line classes, class discussions and virtual group collaboration. Telecommunications is one of the fastest growing fields of technology today. The declining costs of computer systems has given educators and students a new tool for instruction and learning. The fundamental process of electronic mail (email), moving text from one computer mailbox to another, has three general characteristics. First, it is asynchronous or non simultaneous. Second, it is fast. It can be transmitted in seconds or minutes thousands of miles or just down the hall. Third, it is primarily text based. Although most email systems cannot transmit pictures or sounds (Sproull & Kiesler, 1986), techniques and tests for transfer of video and extensive graphics are becoming increasingly more prominent. The use of e-mail to enhance traditional classroom discussion can be a motivating and enriching experience. Students have perceived the advantage of e-mail to center on the greater convenience of access. A single student can discuss and share experiences with many people anywhere in the world. E-mail also provides an alternate delivery system for classroom materials and information (D'Souza, 1992). E-mail technology can provide the communication infrastructure required for effective independent time, independent space group collaboration efforts. There is great diversity in patterns of e-mail usage. Basic mail functions, such as numbers of mail folders, numbers of distribution lists and sizes of in-boxes can vary by at least an order of magnitude. Individuals also vary in their feelings of control over their mail ranging from completely in control to totally overwhelmed.(Mackay, 1989). The effectiveness of the virtual classroom consisting of software enhanced computer communication is contingent in part on the hardware and software capabilities. As previously mentioned, a successful outcome requires adequate access to the necessary equipment, so that both students and instructor may easily access the facility. Probably the single most important behavioral practice which produces relatively good results in on-line courses is the timely and "personal" response by instructors to questions and contributions of students on-line. (Hiltz, 1993). It is important to evaluate the perceived benefits, constraints, and satisfaction user report with the technology in order to identify potential limitations and opportunities inherent to the implementation of the technology in the remote collaborative environment. 2.3.2 Video Phone Case Study As collaborative learning in the virtual classroom was evaluated, a further case study using the technology of two MCI VideoPhones was performed to investigate how alternative technologies may further contribute to the success of collaborative efforts of remote students. The MCI VideoPhone is slightly larger than an average business telephone and has a 3" color LCD display. The phones work over analog telephone lines, which are currently installed in most houses with telephone service. When speaking to another party with an MCI VideoPhone, video is transmitted at about 8 frames per second. It was anticipated that since the video phones support the visual component of communication for remote students, that the phones would increase engagement and individual gratification for remote students using this form of communication. In preparing this study, we expected the introduction of visuals of the remote students via video phone and/or video tape would change both local and remote student perceptions of and communication with other remote students. Concerning small group collaborative efforts using the video phone, we anticipated that the video phone would enhance the engagement and gratification of the parties using the phones, over other modes of communication. Sharing of information, however, was not an expected strong point of the video phone technology. The resolution of the video phone screen does not support the display of printed documents, charts, or other media often used in face-to-face meetings. It was expected that email would prove to be more supportive for information sharing between group members as it does allow the transfer of data and documents, archiving this information as well. 3. Research Methods 3.1 Class Evaluation The use of a survey instrument was the primary means of collecting data from students taking the course, both remote and local to the University of Maryland, College Park. The fifty-question survey prompted students for information concerning their use of different technologies to communicate, the quality of these interactions, how they felt about their involvement with the virtual classroom, and demographic data on the student population. The majority of the questions (80%) were Likert-style questions in which students gave their responses based on a scale from 1 to 5. For example, question number 2 on the survey: 2. How convenient is your access to a computer terminal for email? convenient inconvenient no access 1 2 3 4 5 0 The remainder of the questions were open-ended to give students the opportunity to express opinions, describe difficulties or successes, and offer suggestions for improvements for future implementations of virtual classroom technology. Specific goals of the survey were to determine the means in which students communicated with each other and their ease of access to needed technologies; who students communicated with and how often; motivations for taking the course and for doing well in the course; how involved students felt in the course, both in classroom discussions and in out-of-class collaborative efforts; and to determine if students felt that their collaborations through different technologies were successful. The results of the survey were used to evaluate how students' experiences contained the previously defined components of successful group collaboration, and to determine if use of the video phone in the case study influenced the students' perceptions of and communication with remote students. The survey was administered electronically to the students about eight weeks into the course. Student participation through electronic mail was a requirement of the course, and the scheduling of activities, task assignments, and status of various class endeavors were routinely managed through interactions of electronic mail. Even students that did not participate frequently in email discussions, did use their email account for class activities to some extent, as exams and assignments were delivered through electronic mail. Response rate from the survey was 92%. Additionally, researchers analyzed data from two other sources, the listserv and class interaction sociograms, to verify student responses to the survey. The listserv data consisted of all activity that occurred on the distribution list of the class. Every time mail was sent to the class, it was recorded. This data was used to determine the prominent activities that occurred in class mailings, how participation was distributed among class members, and to serve as a validity check for reported use. In-class interaction was recorded using sociograms by both two researchers, one in the class and one remote. Each sociogram displayed a layout of the classroom, with an object on the sociogram representing each member of the class, both remote and local. When class members interacted or offered a comment or question, the activity was recorded on the sociogram to measure both the level of involvement and the relationships among students. 3.2 Video Phone Case Study Communication between a remote student and the classroom occurred via two MCI VideoPhones, one at the site of the remote student, the other on the desk of the professor teaching the class. This connection allowed the remote student to be both seen and heard live by the class. When the student had the floor, the class broadcast would display the video phone screen, so both local and remote students could see the student. The case study was limited to two video phones. As such, the phones were used by two students working remotely on a shared project to provide additional insight into how the video phones might change the nature of communication. Students using the video phones kept journals recording their reactions, difficulties, and ideas about the technology. 4. Results 4.1 Survey Support for the Critical Factors of Group Collaboration Of the 25 students taking the course for credit, 23 returned completed surveys, yielding a response rate of 92%. The survey results were coded for analysis with numerical values only; textual answers were evaluated, but not used for statistical analysis. Data was analyzed using the software statistical data analysis package SPSS/PC+ Studentware Plus. Statistical data on the student population was compared for two categories of students, remote and on-campus. Eleven remote students and twelve on-campus students returned completed surveys. The remote students were those who received a videotape of the classes on a 2-7 day delay, as well as three metropolitan area students who viewed the class live at their respective company locations. According to the survey data, remote students in this class are typically older than the on-campus students, with two thirds of on-campus students being under age 29 and about two thirds of remote students being over age 29. Most of the remote students are part-time students with full-time job commitments, while most of the on-campus students are full-time students who do not have a typical 40 hour per week commitment at a traditional job. Appendix A illustrates the actual survey that was used for data collection. Appendix B shows results of the students' responses to the survey; distinguishing between remote and on-campus students' answers. The appendix notes any assumptions made in coding the data for statistical analysis. Summary statistics for each question are also shown for each group. An analysis of the results reported by the remote and on-campus students supports the assumption that the factors required for successful collaboration include: ease of access, sense of engagement, sharing of information and individual gratification. Although problems were reported with sharing of information and remote students' feeling of isolation, our hypothesis was supported by a majority of both remote and on-campus students reporting successful collaboration in the virtual classroom. The following sections summarize and discuss the survey results supporting each critical success factor of collaboration, analyzing the results for the two groups to determine whether each factor was present for remote and on-campus students. For each critical success factor, a group of questions were chosen to test the presence of the factor being evaluated. Each of the following sections contains a table summarizing the results that indicate the presence of the success factor. Numbers in the table represent the percentage of remote students and the percentage of on-campus students that reported positively for the respective question. A positive response is interpreted as an answer which is greater than neutral on the Likert scale for positively worded questions and less than neutral for the negatively worded questions. 4.1.1 Ease of Access Survey Responses Supporting Ease of Access Remote Students On-campus Students (% of remote students) (% of on-campus students) Previous Experience with E-mail 42% 84% Convenience 64% 100% Efficiency 36% 83% No Technical Difficulty 82% 58% Since e-mail interaction was a vital means of communication in CMSC 828S, the ease of access to computer terminals was critical to participation and successful collaboration. A majority of remote students reported having convenient access to a terminal, while on-campus students had an advantage over remote students in that they all reported convenient access to computer terminals. Additionally, most on- campus students reported having efficient hardware and software for e-mail, as well as having needed to use an e-mail system prior to this class. Students who reported better convenience to a computer terminal for e-mail, reported that they more frequently checked their messages than did the students without the same convenient access. Although access was better for on-campus students, they reported having technical problems more frequently in using e-mail than did the remote students. Further analysis of students' explanations of problems suggest that many of the on-campus students had problems in receiving information from the remote students. Access to classroom discussion was a problem for the remote students. The only means of participation in classroom activities for remote students who viewed the class on a tape delay, was to send in a videotape. Video-tapes were sent to class by only three remote students. Additionally, two remote students came to class in person once during the semester, and a third remote student used a video phone so that she could more easily participate in class discussions. Even the three students who viewed the class live and had call-in access to the classroom discussion, infrequently attempted such communication even though two of the three students reported being generally active in other classes. One student described the remote access making them "feel like an outsider peeking in with e-mail". Other measures of ease of access showed no significant affects. Specifically, English was not the most effective language for communication for 25% of the on-campus students this did not seem to affect their ease of access or level of participation in this course. Overall, on-campus students reported easier access for this course. Although these differences were found, all students had e-mail access to all other students thus having the opportunity for group collaboration. No student indicated having problems in all aspects of ease of access such as convenience, efficiency and previous experience. 4.1.2 Sense of Engagement Survey Responses Supporting the Sense of Engagement Remote Students On-campus Students (% of remote students) (% of on-campus students) Sense of Involvement 55% 58% Check e-mail greater than four times per day 64% 42% Feeling part of class during e-mail discussions 36% 92% Sense of knowing a remote student 45% 42% Attentiveness to lectures 64% 83% Received timely responses through e-mail 64% 33% Feeling of equality in group contributions 73% 75% Formed good working relationships with students 64% 75% For remote students, e-mail was the best means of class participation. Remote students reported checking e-mail messages more frequently than the on-campus students. Class notes were often posted to Listserv by an on-campus student after the class met, thus partially compensating for the tape delay for most of the remote students. This was seen very positively by remote students and allowed them to participate in class discussions on e-mail regarding issues that had come up during class. Our virtual classroom was successful in stimulating participation equivalent to that in the students' other classes. Only four students in the class, all remote students, felt that they participated less in this class than in other classes. That is, most of the remote students, and all of the on-campus students felt that they were at least as active in this class as in other classes with 30% reporting being more active in this one. It should be noted that CMSC 828S was designed to require significant participation and collaboration. An integral part of this participation was done via e-mail. Question 22 on the survey asked, "During internet discussions, did you feel as though you were a part of the class?" This question was focused on the collaborative environment as supported by electronic mail, where the on-campus and remote students have a more equal opportunity to be included in the virtual classroom. The majority of remote students, however, reported that they did not feel like part of the class even during internet discussions. Thus e-mail interaction did not compensate for the remote students' feelings of isolation. The video phone, however, did further enhance the sense of engagement. Students reported feeling as though they knew the remote student on the video phone better than other remote students. (See video phone section below.) This may be caused by the traditional idea of a class being a group of students in the same place and at the same time. For small group collaborations, on-campus students reported receiving less timely responses from other group members than did the remote students. These results may be explained by a potential difference in expectations of the remote student. Many remote students had become accustomed to the delay in viewing the classes, being able to watch tapes of class at their leisure without their business schedules or travel schedules interfering. In contrast, on-campus students experienced a traditional synchronous knowledge transfer twice a week in class, where they would often see some of their group members. An advantage in synchronous learning may also be improved attentiveness. While none of the on-campus students reported being distracted while watching the lectures, 27% of remote students reported being distracted. This may be explained by the inability of most remote students to interact during the class; viewing the classes in a nontraditional educational environment; and having already been provided a summary of the information discussed during the lecture. One student described feeling "uninvolved, constrained, and bored watching the tapes". Other survey results did not show significant differences in responses between remote students and on-campus students. A sense of engagement was reported very positively in small collaboration groups, remote and on-campus students reported equivalent results in forming good working relationships and feeling that others in their groups had contributed as much as they did. Remote and on-campus students reported a similar sense of involvement and the sense of knowing students which they had never met in person. Students described the group collaborations as "very worthwhile" and "the most positive experiences (in the virtual classroom)". Reports of average time spent on this course ranged from 1 to 27 hours per week with no significant difference between remote and on-campus students. Sociogram Support for Engagement The analysis of class interaction sociograms, recording the in-class discussion observed by both an on-campus and remote researcher, showed an equivalent number of interactions. Although most of the interactions were heard and recorded, the remote researcher was unable to identify an average of 25% of the students who were speaking. This was due to instances that the camera did not follow the speaker and the remote student did not recognize the speaker's voice. The number of unidentified interactions slightly decreased as the semester progressed and the remote researcher became more familiar with students' voices. 4.1.3 Sharing of Information Survey Responses Supporting the Sharing of Information Remote Students On-campus Students (% of remote students) (% of on-campus students) Communicate more due to e-mail 64% 83% Felt negligible constraint in communicating 27% 75% Felt that ideas were successfully heard by others 64% 92% Easy schedule coordination 0% 16% Felt that group was successful in sharing information 55% 73% Although, remote and on-campus students reported no differences in the analysis of information sharing, this factor proved to be the most problematic in our virtual classroom. Even without the face-to- face interaction experienced by on-campus students, remote students reported similar results to on-campus students in the number of classmates with whom they individually corresponded. However, there were differences in the medium used for sharing of information reported by remote and on-campus students. Remote students reported a greater use of telephone calls and fax transmittals, which may be due to the lack of face-to-face, synchronous communication. Remote students reported less than one face-to-face meetings with classmates, whereas 42% of on- campus students reported 1-5 face-to-face meetings with classmates outside of class. About half of the remote students reported 1-5 telephone calls per week and a total of 1-5 fax transmittals. Only one of the on-campus students reported using these media for communication. Media of communication may also have been a factor in the forming of collaboration groups. An analysis of the group members of the EVE and JOVE projects showed that on-campus students tended to work with on-campus students, and remote students tended to work with remote students. For these small group collaborations, all students reported significant problems coordinating a schedule of work, with the remote students reporting more difficulty than the on-campus students. Other small group collaboration differences were found. While on- campus students reported feeling as though their ideas were successfully heard by other members in their group, about one- third of remote students did not report the same feeling. This may be explained by the asynchronous nature of e-mail communication, including the lack of immediate feedback, visual cues, and expressiveness of written communication. Although e-mail discussions allowed all students access to the instructor and one another, the results show a variation in the students' feelings of constraint in communication with the professor or fellow classmates, either remote or on-campus. While about half of the remote students reported feeling constrained, only 8% of the on-campus students reported such feelings. These differences may be explained by the limitations the remote student faces in class participation, the delay in viewing classes, as well as the absence of a sense of knowing the other students. Overall, all students reported some success in sharing information and facts between the group members. However, less than one- fourth reported always being successful at sharing information. Remote students were only slightly less positive in reporting their success. Problems were specifically related to scheduling and remote students' feelings of constraint. 4.1.4 Sense of Gratification Survey Responses Supporting Gratification with Collaborations Remote Students On-campus Students (% of remote students) (% of on-campus students) Found e-mail discussion beneficial 64% 42% Did not feel information overload 36% 75% Would take another class requiring e-mail interaction 73% 67% Communicate more as a result of e-mail interaction 64% 83% Amount of time spent was worth amount learned 73% 50% Feelings of equality in contributions 73% 75% Good working relationships with group members 64% 75% Felt that group collaboration enabled higher quality projects 50% 64% In general, students found CMSC 828S to be satisfying and worthwhile. The majority of both remote and on-campus students agreed that they would take another class requiring e-mail interaction and video technology. Most students felt they communicated more as a result of e-mail. Most students also reported that they formed good working relationships with group members, felt equality in their contributions, and felt that working in groups enabled them to produce higher quality projects. Other indicators of gratification show differences among remote and on-campus students. While information overload was not usually a problem, differences between remote and on-campus students were seen in how frequently students felt overloaded with information. Most on-campus students reported seldom feeling overloaded with information, and most remote students reported sometimes feeling overloaded with information. It may be expected that the more frequently one checks their e-mail messages, the less overloaded they would feel since fewer messages would accumulate. However, results do not show a correlation between frequency of checking e-mail and feelings of information overload. Thus, it seems that there is something more than the volume of messages that contributes to feelings of overload by remote students. One remote student suggested the feeling of overload may be due to having to keep up with a full-time job with overtime in addition to the daily classroom discussion. Another satisfaction indicator tested was the feeling that the class was worth the time spent on it. As noted above the average number of hours per week that students reported having spent on this class varied from 1 to 27. All students reported neutral to favorable responses that the amount of time spent on the course was worthwhile, with remote students reporting slightly more favorably. There was no correlation in the time spent for the course to the student's feeling that the time spent was worth what they got out of the course. Satisfaction in this course was also reported for e-mail interaction as found through successful sharing of information. Remote students found e-mail discussions somewhat more beneficial than on-campus students. As noted above, class notes were often posted to the Listserv which were found to be valuable to the remote students. We suspect that on-campus students' feelings of not benefiting from e-mail discussion was because much of the information was already discussed in class. 4.2 Analysis of VRTP Listserv Mail Electronic mail sent to the class listserv between the period September 10 November 6, 1993 was examined to gather objective information concerning activities within the class as well as to provide a measure to validate student reported activity. During this period, there were 196 messages sent to the group by 28 individuals, including 24 of the 25 individuals enrolled in the class, the professor, the teaching assistant, and one student who was auditing the course. Approximately half of the messages posted (49%) were posted locally by students enrolled in the class, 33% by students enrolled remotely, and 18% of the messages were submitted by the professor and the teaching assistant. The average number of messages posted to the listserv during the period was 22, but the majority of the messages (60.2 %) were posted during weeks two to four of the class when much of the scheduling and assignments for the EVE class project was underway. Other patterns revealed by the data show that most messages are sent Monday, Tuesday, and Thursday, but this is not significantly more than for other days (except weekends). Most messages were sent out to the group without a reply or follow-up (71 of the 196); but 24 messages received at least one reply indicating some dialogue taking place to accomplish goals or to share information. The topical breakdown of the messages according to the criteria described by Sproull and Kiesler (1991) include: Task Number of Messages % of Total Scheduling 23 11.7% Task Assignments 95 48.5% Reporting Accomplishments 29 14.8% General Awareness 49 25.0% Analysis of these messages reveals no significant difference between the number of listserv messages posted by on-campus students and the number posted by remote students, although the total was slightly higher for on-campus students. This difference may be explained in part by the fact that due to the delayed broadcast for some remote students, students enrolled locally sent messages to share information concerning project due dates and other issues which required attention. The results of the listserv analysis were matched with student reports of electronic mail use as collected from the surveys to determine if there were correlations, or at least patterns, present. The data suggest that individuals who posted more messages to the listserv were more likely to report a higher number of individual contacts with other students (6-10 versus 1-5) and a higher level of involvement in the course, but the difference is not significant. Students who reported frequently experiencing overload and technical difficulty sent the fewest number of messages, whether they were located remotely or locally; while students who reported a high level of involvement also reported devoting more hours per week to the class and sent more messages. Student Involvement and Participation (dropouts) Three students who were enrolled in the course dropped the course a few weeks into the semester. These students were contacted by electronic mail and asked to assist in the course evaluation by indicating reasons for leaving. The reasons given include the amount of work required which was not directly related to student research interests and degree requirements which resulted in selection of another class. 4.3 Results of Video Phone Test Trials 4.3.1 The State of Video Phone Technology The MCI VideoPhones used in the case study support a video display rate of 8 frames per second when two parties with the phones are communicating over normal bandwidth telephone lines. As a comparison, video that is displayed on your television is displayed at about 60 frames per second. Thus, the image does not appear to move in real-time and looks somewhat jerky when there is extreme movement. The quality of the picture is fair; you can see the facial expressions and gestures of the remote party well; printed material and detailed objects, on the other hand, are very difficult to see, if you can see anything at all. Popular Science did a study on the AT&T Video Phone 2500, a phone comparable to the MCI model we used, and determined that any text under 1 inch in height was not able to be seen through the video phone (M.M.S., 1993). Due to the slow display rate of the video, there is a problem of synchronizing the audio and video signals. The MCI model that we used put in an intentional voice delay (of about two seconds) so that the sound would more closely match the video. The AT&T phones, on the other hand, do not have such a delay in sound; thus, the sound comes through at speeds comparable to a normal telephone conversation, and the video lags behind. There is a settable option on the MCI phone that allows the user to hear sound without the added delay; this option slows the display rate of the video to less than 8 frames per second. Both the MCI and AT&T phones are quite susceptible to noise on the phone lines; such extraneous data can cause the video image to be lost. We informally experimented with the MCI phone to see just how much data the phone could handle. One party talked and moved as fast as possible, the result being that the video phone displayed an extremely jerky image, while the sound was degraded into only one constant, high pitch. 4.3.2 Video Phone Usage in the Virtual Classroom Participating in classroom discussions via video phone turned out to be much more cumbersome than anticipated. It was thought that the communication would be fairly simple; the remote student would watch the class broadcast in real-time over television, joining in at any time by speaking up. The student could then be seen and heard by the other members of the class. The main reason the experience was not this simple was because of an echo effect heard by the remote student. As the remote student spoke into the video phone, the class would hear the comments with a small time delay of about two seconds. The remote student would continue to speak, but approximately two seconds later, an "echo" of what the student previously said was broadcast through the television, distracting the student, who was by this time, mid-sentence in the next thought. This was distracting, but the student developed work-arounds. To communicate via video phone, it was necessary to combine video phone and TV with a remote control device and quick reflexes. When the student spoke, the TV was muted to eliminate the echo effect. As soon as the student had finished the thought, the volume on the TV was raised to hear the responses of the class. Although this solution did allow the student to speak coherently, there were some undesirable effects. Because of the time delay in the sound reaching the class and because of the muted conditions, the student tended to speak in longer sentences than would be spoken in a same-room, face-to-face conversation. When the remote student was speaking, comments made in the classroom could not be heard because of the muted television. Thus, interruptions and quick exchanges that occur normally in face-to-face discussions did not take place. This effect was particularly notable during a class presentation given by the remote student through the video phone. The class had been made aware of the work-around -- that while the speaker was talking through the phone, any comments made in the classroom could not be heard by the speaker. It was agreed that if someone needed to make a comment during the discussion, then a visual signal, such as someone waving their hand across the screen of the video phone, would be used to alert the speaker of the comment. This plan did not work well because of misread cues when the speaker saw something visual happen and stopped speaking, only to learn that it was an inadvertent motion made by someone on the other side. The majority of the time, the video phone was used in its default mode that incurred the voice delay, so that researchers could attempt to determine the impact of the added visual cues on remote collaboration. It should be noted, however, that the phones do support a faster sound transmittal at the cost of further degraded video resolution. A less frequent, but equally disruptive occurrence was the loss of video while connected. It occurred sporadically and for no apparent reason, most likely noise on the phone line. When someone in the class noticed, the flow of discussions was disrupted as either the professor or a student would attempt to re-establish the link. The quality of the video image seen on the MCI video phone is also quite sensitive to lighting conditions in the room. The resolution of the image degrades sharply when light decreases or is not constant. The message "Too Dark" was frequently seen on the bottom of the video phone's display. To increase the quality of the picture transmitted to the classroom, the remote student set up additional lights for the bi-weekly classes. Admittedly, this is not difficult, but it contributes to the tediousness of the communication. 4.3.3 Small Group Collaboration with Video Phones To evaluate video phone technology as a means to work together remotely, video phone communication will be compared with other available methods for working together from remote sites, namely email and normal telephone communication. Video phone usage as a means to work together remotely has both advantages and disadvantages over email communication. While the asynchronous nature of email communication can be seen as an advantage for getting people's ideas together who are independent of time and/or location, it can also increase the amount of time it takes to resolve an issue, as people to read and respond to messages at their convenience. Furthermore, relative urgencies and importance of issues may not be felt by a reader of an email message, which may impact the timeliness of the reader's response. The advantage of video phone communication, or any real-time communication, therefore, is the immediate communication that occurs. Both parties are focused simultaneously on the problem at hand, allowing resolution and progress to be made much quicker than with several email exchanges. Email, however, provides many benefits not currently supported in telephone or video phone communication. Email provides an immediate and reliable way to exchange data, text, programs, and ideas, and archives the exchanges as well. Email also enables multiple parties to participate in a discussion by allowing messages to be mailed to many people at the same time. A comparable real-time equivalent to this would be conference calling for telephone communication, although it should be noted that the availability of this feature is not yet commonplace outside the business world. Video teleconferencing is another real-time alternative for multiple party communication, but the costs of the technology are limiting, especially to academia. The MCI video phones currently do not support multiple party communication. For the two party communication of our case study, however, we noted that participants felt even more focused when speaking via video phones than with normal telephone conversation. Seeing the person you are conversing with reinforces the engagement of the conversation. You can't easily divert your attention to other things when someone is visually focusing on you. This is a definite advantage of video phones over normal telephone communication. As one participant put it, "Long periods of silence didn't bother me, as I could look up and see if she (the other remote student) was writing, thinking, or waiting for an answer from me. That was comforting, as was the progress we made.". A disadvantage of the video phones, however, is that conversations tended to be substantially longer because of the waiting involved in the sound delay of the phones. The sound delay of the video phones also changed patterns of speaking, which at first was very disorienting. It is very similar to speaking on the phone with someone who is overseas; one party will hear silence and will begin to speak. The silence, however, was just the transmittal time of the audio of the other party, and soon after one party begins to speak, the sound from the other party arrives, effectively interrupting conversation. After a few minutes, most people adjust their speaking, becoming more patient and waiting for sound to arrive. This was the case with the video phones. Since the sound delay of the video phones does not readily support quick exchanges, we found that participants inadvertently modified their speaking patterns to speak in longer sentences, rather than in fragments, as are common in face-to-face, real-time conversation. 4.3.4 Video Phone Conclusions Overall, the sound delay was the most troublesome quality of the MCI video phones, sometimes overshadowing the potential benefits the added video could bring to a conversation between remote parties. Thus, for small group communication, we would prefer normal telephone conversation or video phones without the sound delay, as work done through the medium gets done in real-time. From our experiences using the video phones, we strongly believe that real-time sound is essential to timely discussion and problem solving. However, the use of the video phones to communicate with the virtual classroom was quite successful in increasing the sense of engagement for both the in-class and the remote student. Both remote and in-class students felt as though they knew the remote student on the video phone better than other remote students, even if they had seen the other remote students on video. More importantly, the video phone provided immediate and visual communication of the remote student with the classroom. Because of the visual channel, the remote student on the video phone felt more a part of the class during class discussions than other remote students, even though they could call in and be connected by voice only. In fact, the call-up, audio only nature of this communication for other remote students of CMSC 828S who saw the class live proved to be discouraging. Students felt that calling in was a "big deal" and weren't comfortable with doing it, unless the comment they had to make was important or profound. Calling the class was confusing and disorienting, compared by one student to be like entering a black hole; uncertain of who would answer and how the connection would be made. In fact, one remote student called up the class and was on hold for ten minutes before the student realized that she had been connected to the classroom the entire ten minutes. By that time, the point was mute. Perhaps all that is lacking in this situation is are clear procedures for remote students to communicate with the class. On the other hand, the student on the video phone could easily offer comment or question, without these constraints. The use of the video phone between a remote student and the virtual classroom improved communication by providing more immediate communication, and more communicative interaction by the addition of a visual channel. The limited bandwidth of communication channels and compression algorithms, however, are seen by users as poor quality video and audio, which may reduce its support and/or acceptance for classroom communication. The addition of elements that provide capabilities that are more than what you get in a face-to-face meeting, however, such as the ability to record and transmit data, would even further its potential for success in such an environment. Even with its limitations, the potential benefits of the medium for remote collaboration are encouraging. 5. Conclusions Distance education is successful when the tasks and activities are suited to the technology, are accessible, convenient, and relevant to students, and provide maximum interaction. In this study, we have chosen to emphasize the student experience in the distance learning environment designed to make use of these appropriate tasks and technologies. We identified the following four critical success factors for student collaboration in this environment: ease of access, which includes the convenience and accessibility to the technology; a sense of engagement in class instruction and group projects; the sharing of information; and individual gratification or personal satisfaction from the experience. Our hypothesis assumes that if these factors are present, students will report successful collaboration, and there will be no difference in results based only upon whether students are enrolled remotely or on-campus. These factors were assessed through student surveys, analysis of class interaction, analysis of email activity on the class listserv, and a video phone case study. Critical Success Factors Students responded most positively to survey questions which measured ease of access and sense of gratification. Evidence from both the survey and the listserv analysis point to the convenience of electronic mail and particularly to its value in communicating time-sensitive information to remote students who received class instruction by delayed video. Students reported they communicated more because of electronic mail, and most indicated they would take another course which uses email. Response to the questions related to a sense of engagement in the class was more mixed. In questions directly related to participation, with the exception of four remote students receiving delayed tapes, students reported they participated at least as much in this class as other classes, and nearly a third indicated they participated more. This is significant and consistent with the professor's stated goals and objectives. Much of this sense of engagement and participation was due to the use of electronic mail which encouraged and included the involvement of all students, the video phone and video tapes which provided means of bringing remote students directly into the class, and because of the small group collaborations. Students reported positive experiences from working on an active team on a common goal. One student reported amazement at the ability to develop such good working relationships with remote partners without ever meeting them face-to-face. The down side of the sense of engagement is indicated by a sense of isolation reported by remote students in feeling left out and bored by the tapes where time was spent discussing project tasks or topics that were old news by the time they reached the students. The majority of both the remote and on-campus students reported a sense of knowing the students in the classroom better than the remote students seen through video. Seeing a remote student on video did not change most students' reported perceptions of those students significantly. Additionally, one on-campus student remarked that "working with a remote student allowed me to procrastinate more than I would have" in project development, a case of "out of sight, out of mind." The sharing of information may prove to be the most critical and most problematic of all the factors. Communicating ideas successfully is difficult in the best of circumstances. Misunderstandings are common even when a host of non-verbal cues are present to aid in expression and understanding. Within the virtual classroom, this poses problems for the instructor as well as students. The delayed broadcast and the limitations of the technology inhibited the free and timely exchange of information in some instances. Nearly all individuals indicated problems in coordinating schedules, and some reported difficulty in communicating ideas via the medium. Evidence suggests that the two-way video and audio exchange offered by the video phone as well as student-produced videos helped to establish relationships and to facilitate communication with these individuals. The activities created for this class were designed to stimulate participation and communication to produce something meaningful. According to these intended, first-level effects as defined by Sproull and Kiesler, results were mixed. Most students reported developing good relationships with group members, and many indicated they felt the collaboration produced higher quality projects. For those who did, communication and the sharing of information was clearly a key. One student indicated the communication was essential in "building up a sense of trust with those who demonstrated a willingness to work hard and create a good product." Another described the "motivation to accomplish things" and the "attention focused on the development process, and less toward the personal agendas that usually permeate group projects." Perhaps the most obvious and most important observation is that certain kinds of information-sharing are better for some technologies than others. One student remarked that "having remote students on the team make it difficult to have many faceto-face meetings where most of the work occurs...no amount of electronic mail or other forms of communication can make up for someone standing next to you holding the wires that should be soldered." Limitations of the Study The results of this study tend to support the notion that the elements of successful collaboration are not restricted to same time/same place limitations, but can be approximated by new technologies and communications other than face-to-face. This view is critical to developing successful distance learning programs. However, it is important to indicate some of the limitations of the project. The study was itself an experiment in collaboration, with many of the same problems and issues facing all other groups. In some instances, we reported difficulty in expressing ideas, in obtaining responses, and in coordinating schedules and tasks. Some of the tasks required to produce this paper include development of the survey instrument, deciding valid data collection mechanisms, and apportioning the work. Each of us was located at a different site and each of us had many other responsibilities. Consequently, there were times when deadlines dictated the final product and required that decisions be made under less-than-perfect conditions. Analysis of the data emphasizes some of the less-than-perfect results. One example is in a number of questions that produced confusing results or inconsistent information because they were poorly worded or sequenced. The level of granularity of the Likert scale (1 to 5) proved to be too general, and some respondents invented interim values which more closely reflected their responses. Field-testing the questionnaire more thoroughly might have identified these problems, but wasn't feasible given time constraints. Coordination of schedules and handling much of the business through electronic mail hindered our progress when individuals were unable to respond over a period of several days. Telephone contact helped, but was not always feasible. We had to periodically remind each other of the project focus and the goals and deadlines of particular assignments to keep the work prominent. Towards the end, as we began sharing drafts of sections, the different perspectives and emphases became more apparent, yet the quality of each draft improved significantly as a result of the suggestions of others. It seemed clear as revisions were shared that a common understanding of goals and objectives had pervaded and resulted in better sections than would have been produced by each of us as individuals. Technology, Technique, and Distance Learning Most of the students in this class reported a successful, enjoyable experience. Near the end of the semester, the professor, who had been actively engaged in private and group communication throughout the course, remarked that he would miss the experience when it ended. Electronic mail, student-produced video tapes, the video phone, and face-to-face meetings with students on-campus and at conferences all over the country brought the professor in closer contact with his students than ever before, and offered students access to the professor and to other students in the class that was before unavailable. By using the technology to explore the technology, the course offered a unique opportunity for inquiry and collaboration. How much of this experience is due to novelty, and how much is due to the benefits of the course philosophy to engage students in constructing something meaningful and useful cannot be determined by this evaluation. However, something unique happened here which may be an unintended, second-level effect of the increased communication and of the medium. There seemed to be feelings of respect, of concern for others, of acceptance that are far from normal. Examples can be found in the comments of two students. The first remarked on the success of the group collaboration, describing "most positive experiences came from working with an active team on a common goal...I really liked how the discussion and information transfer went beyond the traditional instructor to student and became heavily oriented on student to student." This student pointed out that the critical success factor for the experience was that members "acknowledge and respect a person's work." The second comment came from a student who was self-described as shy, but found the willingness to speak up electronically "carried over into the classroom setting." This study raises more questions than it answers, and possibly the most provocative may be the potential of technology, previously associated with "depersonalizing" environments, to enhance and extend the learning experience in local as well as distance education when used effectively. ------------------------------------------------------------------ REFERENCES Anderson, Alan (1991). Video-Tunneling to School, SCIENCE, 253. Barron, D.D. (1987). Faculty and student perceptions of distance education using television. JOURNAL OF EDUCATION FOR LIBRARY AND INFORMATION SCIENCE, 27, 257-271. Bavelas, J.B. and Coates, L. (1992). How do we account for the mindfulness of face-to-face dialogue? Communication Monographs, Vol. 59. Bricken, M. (1991). Virtual reality learning environments: Potentials and Challenges. COMPUTER GRAPHICS, 25, 178-184. Bricken, W. (1990). Training in virtual reality. Unpublished paper. (retrieved from anonymous ftp archive ftp.u.washington.edu). Bruce, B., Peyton, J.K., and Batson, T. (eds.) (1993). NETWORK-BASED CLASSROOMS: PROMISES AND REALITIES. Cambridge University Press : Cambridge, UK, 302p. Bulkeley, P.Z., Whittier, D., Osterndorf, W., Baddeley, C., and Yuan, X. (1992). THE ATTRACTIVENESS OF INTERACTIVE COMPRESSED VIDEO IN DELIVERING ENGINEERING EDUCATION TO THE WORKPLACE. A report to the Alfred P. Sloan Foundation, 21p. Cahouet, Pamela (1993). Marketing Research on the MCI Video Phone. Carroll, J.R. (1993). Training via videoconferencing: case studies and usage reports. TELECONFERENCE: THE BUSINESS COMMUNICATIONS MAGAZINE, 12, 27-29. Chapanis, Alphonse (1975). Interactive human communication. SCIENTIFIC AMERICAN, 232, 36-42. Chapanis, Alphonse (1978). Interactive human communication: A few research answers for a technological explosion. Address given at the Eighty-sixth Annual Convention of the American Psychological Association. Toronto, Ontario, Canada. August 1978. Dennis, A.R., George, J.F., Jessup, L.M., Nunamaker, J.F., Vogel, D.R. (1988). Group Decision Support Systems: Concept and Implications. University of Arizona. Unpublished paper. Dennis, A.R., George, J.F., Jessup, L.M., Nunamaker, J.F., Vogel, D.R. (1988). Information Technology to Support Electronic Meetings. MIS Quarterly, Vol. 12, No. 4, pp. 591-624. Dourish, Paul and Bly, Sara (1992). Portholes: Supporting Awareness in a Distributed Work Group. SIGCHI 1992 Conference Proceedings. 541-547. D'Souza, P.V. (1992). E-mail's role in the learning process: a case study. JOURNAL OF RESEARCH ON COMPUTING IN EDUCATION, 25, 254-264. Egan, M.W., Ferraris, C., Jones, D.E. and Sebastian, J. (1993). The Telecourse experience: a student perspective. EDUCATION AT A DISTANCE JOURNAL, 7, J1-7. Ensor, J.R., Ahuja, S.R., Connaghan, R.B., Pack M., and Seligmann, D.D (1992). The Rapport Multimedia Communication System. SIGCHI 1992 Conference Proceedings. 581. Faibisoff, S.G. and Willis, D.J. (1987). Distance education: definition and overview. JOURNAL OF EDUCATION FOR LIBRARY AND INFORMATION SCIENCE, 27, 223-232. Fish, R., Kraut, R., and Root, R. (1992). Evaluating Video as a Technology for Informal Communication. SIGCHI 1992 Conference Proceedings. 37-47. Gallupe, R. B., McKeen, J.D. (1990). Enhancing Computer-Mediated Communication: An Experimental Investigation into the Use of a Group Decision Support System for Face-to-Face Versus Remote Meetings. Information & Management, Vol. 18, Iss. 1, pp. 1-13. Garten, E.D. and Hedegaard, T.C. (1993). Computer conferencing technology and online education: new challenges for assessment and evaluation. EDUCATION AT A DISTANCE JOURNAL, 7, J8-12. Gaver, W., Moran, T., MacLean, A., Lovstrand, L., Dourish, P., Carter, K., and Buxton, W. (1992). Realizing a Video Environment: EuroPARC's RAVE System. SIGCHI 1992 Conference Proceedings. 27-35. Hiltz, S.R. (1993). Correlates of learning in a virtual classroom. INTERNATIONAL JOURNAL OF MAN-MACHINE STUDIES, 39, 71-98. Hiltz, S.R. (1986). The virtual classroom: using computer-mediated communication for university teaching." JOURNAL OF COMMUNICATION, Spring 1986, 95-104. Hiltz, S.R. and Johnson, K. (1990). User Satisfaction with Computer-Mediated Communication Systems. Management Science Vol. 36, No.6. Hiltz, S.R. and Meinke, R. (1989). Teaching sociology in a virtual classroom. TEACHING SOCIOLOGY, 17, 431-446. Hiltz, S.R. and Turoff, M. (1993). THE NETWORK NATION. MIT Press: Cambridge, Mass, 530p. (Chapter 14, p. 471-479). Hiltz, S.R. and Turoff, M. (1985). Structuring computer-mediated communications systems to avoid information overload. COMMUNICATIONS OF THE ACM, 28, 680-689. Hollan, Jim and Stornett, Scott (1992). Beyond Being There. SIGCHI 1992 Conference Proceedings. 119-125. Horowitz, H.M. (1988). Student Response Systems: Interactivity in a Classroom Environment. Unpublished paper. Jessup, L.M., Tansik, D.A. (1991). Decision Making in an Automated Environment: The Effects of Anonymity and Proximity with a Group Decision Support System. Decision Sciences, Vol. 22, Iss 2, pp 266- 279. Kozak, J.J. (1993). Transfer of training from virtual reality. ERGONOMICS, 36, 777-784. Kraut, R., Egido, C., Galegher, J. (1889). Patterns of contact and communication in scientific research collaboration. Unpublished research paper. M.M.S. (1993), Electronic Newsfront, POPULAR SCIENCE, 30-45. McClelland, A. (1993). E-mailing a large class. IT TIMES, 1, June 1993. Retrieved from the Internet. Olsen, Bengt (1992). Third-generation distance education. International Journal of Continuing Engineering Education, Vol. 2, Nos 2/3/4. Olsgaard, J.N. (1987). The Impact of a distance education program on enrollment patterns. JOURNAL OF EDUCATION FOR LIBRARY AND INFORMATION SCIENCE, 27, 272-279. Osterndorf, W., Baddeley, C., Yuan, X. and Whittier, D. (1992). Interactive compressed video: the evolution of distance education. Research report, 86p. Pea, R.D and Gomez, L.M. (1992). The Collaborative visualization project: shared-technology learning environments for science learning. In PROCEEDINGS OF SPIE '92. (International Society of Photo-Optical Instrumentation Engineers). Sanderson, Ian (1993). Desktop Video '93. TELECONFERENCE, 18-37. Sellen, Abigail (1992). Speech Patterns in Video-Mediated Conversations. SIGCHI 1992 Conference Proceedings. 49-59. Sellen, Abigail and Buxton, Bill (1992). Using Spatial Cues to Improve Videoconferencing. SIGCHI 1992 Conference Proceedings. 651-652. Shavelson, R.J., Stasz, C., Schlossman, S., Webb, N., Hotta, J.Y. and Goldstein, S. (1986). EVALUATING STUDENT OUTCOMES >FROM TELECOURSE INSTRUCTION: A FEASIBILITY STUDY. Rand : Santa Monica, Ca., 63p. Research Report R-3422-CPB prepared for the Corporation for Public Broadcasting. Shneiderman, B. (1993). Education by engagement and construction: experiences in the AT&T teaching theater. Keynote speech for ED-MEDIA '93, June 1993, Orlando, Florida. Silverman, B.G., Boehm-Davis, D., Norman, K., and Sebrechts, M.M. (1993). Collaborative distance learning in the consortium: designing a course in human-computer interaction. A proposal submitted to Cooperative Grants for Faculty Consortium of Universities of the Washington Metropolitan Area. Sproull, L. and Kiesler, S. (1991) CONNECTIONS: NEW WAYS OF WORKING IN THE NETWORKED ORGANIZATION. MIT Press : Cambridge, Mass. 212p. Takemura, Haruo and Kishino, Fumio (1992). Cooperative Work Environment Using Virtual Workspace. CSCW 1992 Conference Proceedings. 226-232. Turoff, Murray (1991). Computer-mediated communications requirements for group support. JOURNAL OF ORGANIZATIONAL COMPUTING, 1, 85-113. Wilmeth, R. (1993). The Focus on the teacher. TELECONFERENCE: THE BUSINESS COMMUNICATIONS MAGAZINE, 12, 14-17. Yager, Tom (1993). Better Than Being There. BYTE, March 1993, 129-134.