Presence and Communication in Hybrid Virtual and Augmented Reality Environments

Abstract The use of virtual reality (VR) and augmented reality (AR) in connected environments is rarely explored but may become a necessary channel of communication in the future. Such environments would allow multiple users to interact, engage, and share multidimensional data across devices and between the spectrum of realities. However, communication between the two realities within a hybrid environment is barely understood. We carried out an experiment with 52 participants in 26 pairs, within two environments of 3D cultural artifacts: (1) a Hybrid VR and AR environment (HVAR) and (2) a Shared VR environment (SVR). We explored the differences in perceived spatial presence, copresence, and social presence between the environments and between users. We demonstrated that greater presence is perceived in SVR when compared with HVAR, and greater spatial presence is perceived for VR users. Social presence is perceived greater for AR users, possibly because they have line of sight of their partners within HVAR. We found positive correlations between shared activity time and perceived social presence. While acquainted pairs reported significantly greater presence than unacquainted pairs in SVR, there were no significant differences in perceived presence between them in HVAR.


Introduction
Whilst collaborative virtual environments (CVE) research spans a lengthy history and has benefited users with shared experience in symmetric environments, shared experience in immersive virtual and augmented realities can be very different. In the present research, we explore the concept of asymmetric interactions in Hybrid VR and AR environments (HVAR) with the goal of connecting users between the different realities. We are motivated by the potential of immersive environments and the affordability of mobile devices that can support real-time 3D displays. It is also an increasing trend toward crossplatform collaborations with VR and AR technologies (Lee & Yoo, 2021;Speicher et al., 2018). Research has shown that some users are more susceptible to VR induced symptoms and effects (Sharples, Cobb, Moody, & Wilson, 2008). Although improvements with immersive display technology will reduce such effects, the hybrid use of VR and AR may become necessary to cater for a wider range of needs and scenarios. VR systems tethered to workstations and extraneous tracking sensors can be costly whilst mobile AR or even VR can be an alternative choice for accessing multidimensional data. We believe that HVAR environments could be useful for many application areas that necessitate communication and social interactions, such as public display, education, training, and entertainment. The collaborative use of VR and AR has been demonstrated to be beneficial in supporting task-oriented cooperation, coordination, and information sharing (Billinghurst, Kato, & Poupyrev, 2001;Piumsomboon, Day, Ens, Lee, Lee, et al., 2017). However, research on communication and social interactions within shared social spaces is scarce, and there are no studies on communication within HVAR environments reported in the literature.
In this study, we investigate factors of hybridity between VR and AR. Our study expands on findings from a previous study on the technological acceptance of HVAR environments (Li, Ch'ng, Cai, & See, 2018). We investigate how communication differs between hybrid VR and AR environments in an experiment involving 52 participants of 26 pairs, and evaluate perceived spatial presence, copresence, and social presence. We compare our findings between the Hybrid VR and AR environment (HVAR) and the Shared VR environment (SVR), and between VR and AR users in HVAR. We also measure users' activity data in both VR and AR to calculate users' shared activity time. Shared activity refers to the occasions when users are in close proximity to the same object at the same time.
We begin this article with a review of related work on collaborative VR and AR. Next, we present the experimental design of HVAR and SVR environments, and define research questions and hypotheses that we aim to test and answer. Finally, we present the results from our analysis, discuss implications of our research and conclude our findings.

2
Related Work

Collaborative Use of VR and AR
Collaborative systems can be categorized into four categories, based on Johansen's (1988) classification matrix of the time and space dimensions. Early research on CVE was primarily focused on distributed systems to support synchronous and remote collaboration, such as Carlsson and Hagsand's (1993) DIVE platform for multiuser interactions, Greenhalgh and Benford's (1995) MASSIVE system for teleconferencing communication, and Benford et al.'s (1995) VR-VIBE application to support cooperative work on documents. These CVEs provided users with symmetric experiences and allowed users based in different locations to share information. However, Billinghurst, Weghorst, and Furness (1998) argued that CVEs separate users from the real world, and can be hard to be adapted to users' workspace. Therefore, they explored the collaborative use of AR for synchronous co-located experiences. They introduced the Shared Space concept and described several interaction and visualization techniques for users' shared views in co-located collaboration. In addition, Benko, Ishak, and Feiner (2003) presented VITA, a visual interaction tool that combined various projected interfaces, tracked handheld displays, and large screens for multiuser co-located archaeological excavations. Such use of co-located collaborative AR can take account of the situated contexts in facilitating collaboration, leveraging users' visibility to the real world.
Aside from the symmetric experiences in either collaborative VR or collaborative AR, researchers also explored hybrid use of AR and VR with tabletop interfaces and desktop PCs and have designed asymmetric experiences in collaborative work. Stafford, Piekarski, and Thomas (2006) explored hybrid use of AR (for outdoor use) and a tabletop interface (for indoor use). They presented "God-like" metaphor interaction techniques that enabled two users to work together remotely on locationbased tasks. Duval and Fleury (2009) presented a hybrid use of VR and desktop PC to exploit their respective 2D and 3D features in selection and manipulation tasks. Ibayashi et al.'s (2015) Dollhouse VR demonstrated a colocated experience with a user in VR and two users using a multitouch tabletop, collaborating on the architectural design with different views and interaction styles. These studies illustrated how the hybrid use of various displays and interaction techniques can help create asymmetric user interactions for remote or co-located collaboration. However, the studies used either VR or AR with other technologies; none of them explored the use of both VR and AR in a connected experience. In addition, systems used in these studies were primarily designed for task-oriented collaboration processes with a focus on the cooperation, coordination, and information sharing. The classification of collaborative systems (Andriessen, 2012;Penichet, Marin, Gallud, Lozano, & Tesoriero, 2007) also include communication (person interchange processes) and social interactions (group-oriented processes), which were studied less in previous collaborative VR or AR work.

Hybrid VR and AR Use
One of the earliest examples of hybrid use of VR and AR was Kiyokawa, Takemura, and Yokoya's (2000) SeamlessDesign tool. It incorporated both augmented and virtual environments for collaborative creation of 3D objects. The seamless view-mode switching and the multiscale collaboration features of SeamlessDesign can also be seen in Billinghurst et al.'s (2001) Magic-Book, a transitional VR and AR interface with different viewing points, and in Piumsomboon et al.'s (2017) Co-VAR, a collaborative VR and AR system that supported view scale changes for remote collaboration. In addition, Oda, Elvezio, Sukan, Feiner, and Tversky's (2015) work on virtual replicas demonstrated how a remote subject-matter expert could use VR or AR with annotations to assist a local user in AR with physical objects. A recent study conducted by Grandi, Debarba, and Maciel (2019) compared the co-manipulation of objects and task performances with three different VR and AR interfaces. These works demonstrate that the hybrid use of VR and AR can provide unique user experiences and collaborations utilizing their different features in viewpoints, scales, and interaction techniques. However, similar to the symmetric experience in collaborative VR and AR, these systems were primarily designed for taskoriented processes that are concerned with cooperation, coordination, and information sharing, thus focusing on the technological foundations and system development. Gugenheimer, McGill, Steinicke, Mai, Williamson, et al. (2019) argue that current adoption of VR and AR needs to address the challenges of usage in shared social environments and contexts, namely the copresence of others.
They suggest that, in addition to technical foundations and system development, it is vital to focus on the actual use of such environments. We believe that a fundamental element of usable hybridity between VR and AR is communication. Effective communication will support person interchange processes and social interactions in group-oriented processes in the use of VR and AR. These concepts related to communication are also interwoven with presence concepts of which other users' interactions are implied.
The ShareVR (Gugenheimer, Stemasov, Frommel, & Rukzio, 2017a) and the FaceDisplay (Gugenheimer, Stemasov, Sareen, & Rukzio, 2017b) are examples tackling issues in group-oriented social interactions with the use of VR HMD. The ShareVR prototype demonstrated how non-HMD users can be part of the HMD users' experience through floor projections, mobile displays, and positional tracking. The FaceDisplay displayed the view seen by mobile VR users to bystanders and allowed them to interact through touch screens. Such studies show how the inclusion of interactions from non-HMD users within the immersive environment viewed by the HMD user can lead to an increase of enjoyment, presence, and social interaction. However, it is not clear how users in different environments perceive themselves or others in the connected experience. In such an interchange process, perceived presence and communication have not been formally studied. This is especially true when non-HMD users are allowed to enter the virtual space of VR users via AR.
Discussions of social presence often involve copresence in the literature (Ijsselsteijn & Riva, 2003;Nowak & Biocca, 2003;Skarbez et al., 2017;Zhao, 2003). Copresence is a concept grounded on the basic sensory awareness of others, implying the reception of embodied messages and mutual awareness (Goffman, 1959). In other words, copresence denotes both the physical condition, known as the mode of being with others, and the subjective experience of the sense of being with others (Zhao, 2003). Ijsselsteijn and Riva (2003) stated that copresence is the intersection of spatial presence and social presence. It shares properties with spatial presence, such as being in the same place, and the social presence perspective that concerns the awareness of and connection with others. However, Biocca et al. (2003) viewed copresence as a dimension of social presence, although their explanation of copresence also mentioned the spatial relationship between people. Based on Biocca et al.'s (2003) work and Slater's (2009) work on place illusion (the illusion of being there) and plausibility illusion (the illusion that the scenario being depicted is actually occurring), Skarbez et al. (2017) further proposed social presence illusion (the feeling of social presence engendered by characters in virtual or mediated environments) and identified copresence illusion (the feeling of "being together" in a virtual or mediated space) as influencing factors. Both copresence and social presence are user-centric and indicate the subjective experience of users such as awareness, connection, involvement, and engagement, etc. with others in social contexts. Therefore, copresence and social presence are essential factors in the study of the aforementioned subjective perceptions and the communication among people in connected experiences.

Research Questions and Hypotheses
Previous research has identified factors that contribute to presence, including the quality of visual display resolution, interactivity of the environment, users' self-representation, the connection between actions and effects, and internal factors influencing user responses to stimulus in virtual environments (Barfield & Weghorst, 1993;Heeter, 1992;Sheridan, 1992;Slater et al., 1994). These factors are influenced by the characteristics of both media and users, a great part of which can be attributed to systems consisting of hardware and software that provide the visual display, and the more nuanced and subjective perceptions of users. There is certainly a difference in how computing capacity, display size and resolution, and affordances of control mechanisms can shape the perception of users between VR and AR.
Here, we study the social context allowable by communication via the hybridity between VR and AR. We compare the experience of paired users participating in shared activity in one of the two environments: HVAR and SVR, and ask the question: "how does communication differ between hybrid VR and AR environments?" by formulating three subquestions:

RQ1.
Are there perceived differences in presence between HVAR and SVR? RQ2. Are there perceived differences in presence between VR and AR users within HVAR? RQ3. Does shared activity time correlate with perceived social presence?
VR provides users with rich sensations, such as visual, auditory, and haptic stimuli, and can consequently lead to the illusion of being "present" in the simulated place (Mania & Robinson, 2005). However, AR's augmentation of virtual objects in the real environment involves less sensory information. Previous research has found that HMD users in VR reported greater spatial presence compared with non-HMD users (Gugenheimer et al., 2017a). We therefore propose that: H1a. Users in SVR perceive greater spatial presence than users in HVAR. H2a. Users in VR perceive greater spatial presence than the AR users in HVAR.
VR affords a wide array of social cues compared with other forms of computer mediated communication systems (Oh et al., 2018). Avatars have been demonstrated to be helpful in facilitating social interactions (Schultze, 2010). In this research, we propose that: H1b. Users in SVR perceive greater copresence than users in HVAR. H1c. Users in SVR perceive greater social presence than users in HVAR. H2b. Users in VR perceive greater copresence than the AR users in HVAR. H2c. Users in VR perceive greater social presence than the AR users in HVAR.

Hybrid VR and AR Environments: Experimental Materials
We developed a set of environments to allow users to engage in shared viewing and exploration of artifacts in a virtual museum. A VR environment and an AR application were designed in view of our questions on communication in hybrid reality. We first developed a Hybrid VR and AR environment (HVAR) connecting users using high-end workstations and low-end mobile devices. We also developed a Shared VR environment (SVR) that connects VR users in the same virtual space. Our environments were able to host multiple VR and AR users in a co-located experience with access to virtual objects.

Materials
Six close-range photogrammetry 3D models were constructed, processed, and used in system development. Our choice of objects were cultural relics with a mixed origin. Of course 3D objects of other genres were possible, but our view was that virtual objects of cultural relics would sustain user interests much more than contemporary objects. Models of the cultural relics were processed and retopologized in the Blender 3D modeling software, optimized for real-time interactions targeting both workstation VR and mobile AR. Information about the relics was collected from our previous field work (see Ch'ng, Cai, Leow, & Zhang, 2019) and museum websites. Details are shown in Table 1.
System development details are summarized in Table 2. A Wireless Local Area Network was set up to connect users and synchronise user interactions in HVAR and SVR environments. A network lobby was set up to manage the network, including the server setup and client connections. For a shared activity, one user joined the network connection as a host (server and client), and the other user connected to the host using the host's IP address.
In summary, the HVAR connected one user in VR with an HTC Vive headset and two handheld controllers, and one user in AR using a smartphone and a physical AR cube. The SVR connected both users in the same virtual environment, each using a set of HTC Vive headset and controllers.

VR Environment
Within the VR environment, six museum objects were acquired photogrammetrically and rendered with photographic texture. They were placed on top of pedestals and arranged in a circular enclosure The mask is one of the two largest bronze masks unearthed at Sanxingdui. It has very big eyes and ears, which are so exaggerated as to live up to their great power of seeing and hearing from faraway.

Bronze Music Instrument
This oval-shaped percussion instrument is inscribed with 79 characters. The vessel is complete, with exquisite decoration and a sense of imposing majesty.

Xie Zhi (Pottery Unicorn)
The unicorn is a beast that symbolizes justice. Its horn is dedicated to those who are unjust in law enforcement.

Tri-Colored Camel
Tri-colored camel of the Tang Dynasty, hanging a bag with animal's face, silks, and a kettle.

Pottery Figure of a Standing Lady
This female figure is an example of grave goods. The figure displays the realistic style of Tang art, embodying for us the natural appearance of Tang noble women.

Figure of an Assistant to the Judge of Hell
This pale-faced clerk is carrying a slim scroll, recording the few names of those who have performed good deeds in their lives. This figure originally came from a temple and stood either side of a judge of hell.  (see Figure 1). For each object, a label containing an image and texts obtained from museum websites was available in both English and Chinese in view of the demographics of our participants. The information labels were placed in the virtual environment along with the objects. The design of the exhibition room was kept minimal in order to focus the attention of our users on virtual objects. Users were allowed to: (1) walk around freely within the 3.5 m × 3.5 m space, (2) view virtual objects from different perspectives, (3) view the information label of virtual objects, and (4) interact with the objects using both hand-held controllers.
We mapped the navigation inside the virtual environment with users' physical movements in the real world, providing a one-to-one correspondence of the virtual to physical environment. This approach ensured that changes to the direction and relative distance were visible and natural to our users in the virtual environment, thus mitigating the risk of simulator sickness. Users were able to grab objects with both controllers using the trigger buttons. Rotation of objects was achieved using rotation of the controllers (see Figure 2). This allowed viewing of virtual objects from different perspectives, facilitating increased exploration as compared to passive 36 PRESENCE: VOLUME 28  viewing. The original position of the target object was highlighted when the object being grabbed was close to the pedestal. The object snapped back to the original position on pedestal if the user released the trigger button.

AR Application
The AR application integrated with a physical cube comprising a 2D image target on each face (see Figure 3a). The mobile AR recognized the 2D image on the AR cube and triggered the augmentation of the linked 3D model on top of it (see Figure 3b). Once aug-mented, the 3D model rotated on the applicate (z) axis. The objects could be viewed from different angles via the manual rotation of the AR cube. More details can be found in Li, Yu, and Liang's (2021) CubeMuseum prototype.
Information labels for each object were augmented on the right side of the display (see Figure 3b). The information labels reflect the same amount of textual information as in VR: object name, size, time period, affiliated museum, and brief history. The information label was triggered by default but could be dismissed at any time by tapping on it. Labels could also be brought up by tapping on the virtual object. Unlike the VR environment, the AR application had no virtual exhibition room as the physical location in its environment. Virtual objects in both VR and AR could be viewed from different perspectives using the different approaches described above.

Hybrid VR and AR Environment
Within HVAR, we used virtual objects as the interface between VR and AR. The object itself was the connection between synchronized user interactions in HVAR. Virtual object rotations were synchronized in both environments, providing visual cues to inform each respective user that the object was being viewed. Aural cues from sound effects were triggered for both clients if an object was grabbed in the virtual environment or augmented on the AR cube. In addition to the visual and aural cues synchronized through the network, the AR user could see the VR user's first-person view mirrored on the TV. Users were able to converse with each other at any given time (see Figure 4).

Shared VR Environment
Similar to the HVAR environment, interactions in SVR were synchronized over the network. A sound effect was triggered for all users when objects were being interacted with. The difference between HVAR and SVR was that HVAR synchronized rotations of virtual objects, whereas SVR synchronized real-time positions and rotations of the virtual objects.
Each user in SVR had a virtual avatar representation consisting of a simple spherical object which indicated the gaze and two controllers representing the hands (see Figure 5). The virtual avatar was simple in visual style but with clear representation of behavioral realism (Bailenson & Yee, 2006). The synchronization of avatar movements was reflected in realtime for both users. This was an additional feature of SVR that HVAR did not have. Users were able to converse at any time within SVR.

Experimental Methods
We used an established multidimensional approach for evaluating the communication in HVAR and SVR using subjective, process, and performance measures (Kiyokawa, Billinghurst, Hayes, Gupta, Sannohe, et al., 2002). A favorable ethical opinion was provided by the University of Nottingham Ningbo China's Ethics Committee. All participants were paid an honorarium for their contributions to the study. In total, 52 participants (28 male, 24 female) aged 18-54 (M = 25.58, SD = 6.28) were recruited. Participants were students or staff of the university, and their families and friends. Participants could sign up as pairs or as a single user to be randomly paired. Among the 26 pairs of participants, 20 pairs were previously acquainted and the remaining 6 pairs were not. Participants were asked to evaluate their skills in 3D gaming, VR, and AR if they had such experience (see Table 3). Overall, participants who had 3D gaming, VR and AR experiences considered themselves to be reasonably skillful at them.

Subjective Measure: Questionnaires
Although the use of questionnaires to evaluate presence has been contested (Slater, 2004;Usoh, Catena, Arman, & Slater, 2000), subjective questionnaires have been the standard evaluation of presence in the literature whilst physiological measures have yet to be well established (Pike & Ch'ng, 2016;Slater, Guger, Edlinger, Leeb, Pfurtscheller, et al., 2006). Retrospective questionnaires are robust and reliable, and have proven to be adequately sensitive to reveal differences (Insko, 2003). We used the presence questionnaire (Nowak & Biocca, 2003) to evaluate spatial presence (Lombard & Ditton, 1999), copresence (Burgoon & Hale, 1987), and social presence (Short et al., 1976). Table 4 explains the scales of the presence questionnaire. We calculated the Cronbach's alpha (CA) to measure the internal consistency of the psychometric scales, yielding a value greater than 0.70 at all four scales.

Process Measure: User Activity Monitoring
The process measure was inspired by the use of user activity monitoring in analyzing online communities (Lampe, 2013) and the digital nature of VR and AR systems. We implemented functions to record user activity data within the VR environment and with the AR application. Specifically, we recorded VR users' gaze  Nowak and Biocca (2003)

Description
Cronbach's alpha Spatial presence The sense of "being there" in the virtual environment. 0.86 Self-reported copresence Includes items about intimacy, involvement, and immediacy.

0.75
Perceived other's copresence Includes items about intimacy, involvement, and immediacy.

0.89
Social presence Indicates the perceived ability of the medium to connect people.

Table 5. Two Communication Topics Provided to Users and Their Summary
Topics Summary 1 Please identify the object you liked most and explain why.
Ranking based on pairs' subjective preferences.
2 Please rank the historical chronological order of the six objects.
Ranking based on pairs' obtained information and prior knowledge.
information as tracked by the HMD and interaction information as tracked by the controllers. We also captured data when the AR users triggered an augmentation and touch action points on objects and labels. Raw data were stored in a CSV file once the program was shut down. User activity monitoring provided objective measures for user interactions with the VR and AR systems, and enabled analysis of shared activity time in HVAR and SVR.

Performance Measure: Two Communication Topics, Observations and Interviews
Performance measures are standard in taskoriented processes for evaluating task performances, such as measuring the time it takes to complete a task. As part of the communication aspect of our research, we asked users to discuss two topics during their experience (see Table 5). To evaluate their communication outcomes, we asked the participants to provide rankings of the six objects based on their subjective preferences and to see if they were able to identify the correct historical chronological order of the objects.
Later, we combined the process and performance measures, and used observations and interviews to complement our understanding of the communication occurring between the objects and the users. Observation notes were taken during the experiment and a short interview was carried out at the end to discuss and compare experiences in HVAR and SVR.

Setup and Experimental Procedure
The experiments took place at the NVIDIA Joint-Lab on Mixed Reality, an NVIDIA Technology Centre at the University of Nottingham's China campus. Each experiment with paired users lasted for about an hour. Participants were informed that they could remove the headset at any time during the study if they felt any discomfort, but there were no such events. Users were briefed on the study, use of the VR and AR 40 PRESENCE: VOLUME 28 technologies, which included the headset, handheld controllers, the smartphone, and the AR cube. Users filled in a pre-experiment questionnaire on user demographics prior to the beginning of the two experimental sessions. The order of the HVAR and the SVR sessions was counterbalanced: half of the pairs completed the study first in HVAR then in SVR and the other half completed the study first in SVR and then in HVAR (see Figure 6). Users were in the same room for both sessions. During each session, users discussed a given topic, and their activity data at system runtime were recorded. Both users in each pair were required to fill in the presence questionnaire after each session. After the two sessions, a short interview was conducted based on the observation notes taken during the experiment.

Results
Our data samples include responses from the questionnaires, quantitative user activity data collected from system runtime, and qualitative data from observations and interviews. We confirmed parametric test assump-tions and performed t -test analysis to ascertain differences reported in the questionnaires between relevant paired conditions. Specifically, paired-samples t -tests were performed for comparisons between the two sessions: HVAR and SVR; independent-samples t -tests were performed for comparisons between VR and AR users in HVAR. Significance values that we report are one-tailed because our hypotheses were directed. We conducted Spearman correlation analysis to identify the association between the shared activity time and social presence. Results of the hypotheses are summarized in Table 6. User A refers to the AR users in HVAR and user V refers to the VR users in HVAR.
In the following sections, we present the results for the analysis of presence (subjective), the user activity data (process), users' discussions, our observations, and interview data (performance).

Presence Questionnaire
The results of the presence questionnaire for SVR showed significant positive correlations between all four presence scales (see Table 7). For HVAR, all correlations

Hypothesis Result
H1a Spatial presence is greater in SVR than in HVAR Supported H1b Copresence is greater in SVR than in HVAR Supported H1c Social presence is greater in SVR than in HVAR Supported H2a Spatial presence is greater for HVAR-V than for HVAR-A Supported H2b Copresence is greater for HVAR-V than for HVAR-A Rejected H2c Social presence is greater for HVAR-V than for HVAR-A Rejected H3a Shared activity time correlates positively with social presence in HVAR Supported H3b Shared activity time correlates positively with social presence in SVR Supported were significant except for the correlation between spatial presence and self-reported copresence. Figure 7. Spatial presence and social presence were evaluated on a seven-point Likert scale, whereas the self-reported copresence and perceived others' copresence were reported on a five-point Likert scale (see Nowak & Biocca, 2003). A paired-samples t -test was conducted to test the hypothesis that perceived spatial presence is greater in SVR than in HVAR (H1a). The results indicated that perceived spatial presence was significantly higher in SVR (M = 5.55, SD = 0.95) than in HVAR (M = 4.85, SD = 1.09), t (51) = 5.08, p < .001. H1a is supported. Specifically, user A perceived greater spatial presence in SVR (M = 5.64, SD = 0.96) than in HVAR (M = 4.51, SD = 1.10), t (25) = 5.05, p < .001; user V also perceived greater spatial presence in SVR (M = 5.46, SD = 0.94) than in HVAR (M = 5.20, SD = 0.97), t (25) = 2.47, p < .05.

Comparison of Presence Between the VR and AR Users in HVAR.
The comparison of presence as perceived by the VR and AR users in HVAR is illustrated in Figure 8.
An independent-samples t -test was conducted to test the hypothesis that users in VR perceive greater spa-tial presence than AR users in HVAR (H2a). The results indicated that VR users (M = 5.20, SD = 0.97) perceived significantly greater spatial presence than AR users (M = 4.51, SD = 1.10), t (50) = 2.40, p < .05. H2a is supported.
An independent-samples t -test was conducted to test the hypothesis that users in VR perceive greater social presence than AR users in HVAR (H2c). The results indicated that AR users (M = 5.11, SD = 1.07) perceived significantly greater social presence than VR users (M = 4.27, SD = 1.49), t (50) = 2.33, p < .05. Thus, H2c is not supported.
The comparisons of copresence between VR and AR users in HVAR showed no significant difference. H2b is not supported. There were no significant differences shown for the comparison of spatial presence, copresence, or social presence between the paired users in SVR either.

The Acquaintance Effect on Presence.
An independent-samples t -test was conducted to test the hypothesis that acquainted pairs perceive greater presence than unacquainted pairs in SVR. The results indicated that acquainted pairs reported significantly greater spatial presence, perceived other's copresence, and social presence than unacquainted pairs in SVR (see Table 8).
However, there were no significant differences reported for presence in HVAR.

User Activity Data
For every one-second interval of user activity data, we tagged an object as interacted with if gaze was focused on it or the object was interacted with. We recorded the data to analyze users' shared activities, that is, the occasions when users were in close proximity to the same object at the same time. If both users were in close proximity to the same object at one second intervals, the tracked data were counted as shared activity time for paired users. Table 9 summarizes the results of the total time (T ) users spent within each session, the length of shared activity time (T SA ), and the shared activity time ratio (R SA ). The ratio indicates the percentage of time in shared activities, calculated using the formula: Users spent 276.81 seconds on average for each session in HVAR, of which 73.23 seconds (26%) were time in shared activities. In SVR, users spent 345.15 seconds on average for each session, of which 74.23 seconds (21%) were time in shared activities. There were no significant differences shown for the total time, shared activity time, or the shared activity time ratio between HVAR and SVR.
Spearman correlation analyses were conducted to test for positive correlations between shared activity time and social presence (H3). The results were significant for HVAR, r s (52) = 0.34, p < .01, and SVR r s (52) = 0.53, p < .01. Therefore, H3a and H3b are both supported.

Communication Outcomes
Here, we gauge the outcomes of the in-session communication. During the first session, users discussed their subjective preferences with paired partners for the six virtual objects and ranked them between 1 to 6 (1 = least preferred, 6 = most preferred). The total score for each object was calculated by summing all ratings (see Table 10). Users were more interested in the Bronze Mask with Protruding Pupils and the Pottery Unicorn compared with the other objects. This was in line with our observations of users' interactions with these two objects. For example, we observed that VR users attempted to "wear" the mask or attempted to adorn their partners in SVR. Users also used the horn of the Pottery Unicorn as a weapon to "attack" their partners. Users commented in the interview that the significant action possibilities in VR did enrich their experiences compared to passive viewing of objects.
During the second session, the historical chronological orders of the six objects were discussed between the pairs and rankings were provided (1 = the most ancient, 3.85% (1) Pottery Figure  3.85% (1)

92.31% (24)
6 = the most recent). The responses are presented in Table 11 in the correct historical chronological order of the objects from top to bottom. The labels in both VR and AR provided information on the time periods of each object from which users could discuss their answers. Users also combined the given information with their prior knowledge of history and the objects. Based on the answers provided by our participants, the correct rates for each object were all above 75%, which indicated the positive outcomes of user communication between pairs. Participants commented during the interview that the information exchanged during the sessions contributed to their learning about the objects.

Observations and interviews
We observed that users did follow some social norms and mannerisms. While users swapped the positions of the objects in VR, they attempted to put them back in their original positions at the end of each session. They reported in the interview that they did not want to confuse other users. We consider this an aspect of communication that is transferred from the physical world to the virtual environment. In addition, the majority of users in SVR greeted their paired partners by hand-waving, saying "hi" or both. Users also demonstrated attention in their gaze, by looking at their partners' avatars when having a conversation. They reported in the interview that they had more awareness of their partners as they were able to see their actions in the environment.
Although we did not deliberately design collaborative tasks for paired sessions, spontaneous collaborations were observed in both SVR and HVAR. Some users collaborated to memorize the historical chronological order of the objects by dividing the six objects into two groups of three. We also observed a cooperative phenomena from the object Figure of an Assistant to the Judge of Hell. This object was not movable in VR, but the rotations could be triggered by the AR user and be seen in the VR environment. Several pairs took advantage of this asymmetric interaction opportunity and assisted the VR user in viewing the different sides of the object. We also observed one instance of action that was reminiscent of a "guided tour" in HVAR where the AR user guided the paired VR user for each object in a sequential order. The AR user read the information label of the object and explained the object story while the VR user interacted with the object using the controllers and rotated the object for the paired AR user through the mirrored display.
In the interview, half of the participants compared the two VR sessions they had and commented that avatars in SVR were helpful in tracking the presence of the other, compared with HVAR where there was no avatar, although having a sense of social presence in HVAR was better than an isolated session where the participant was the only person in the environment. Some users reported that the inability to override the object held by the AR users could be disappointing, for example, their objects could be affected by the AR user via rotations but this could not be done the other way around. The other half of the participants compared their AR experience in HVAR with VR in SVR. They acknowledged that the full immersion and interactivity allowed in VR was a better experience overall. Others reported that they were more comfortable using AR as they could see the augmented objects and information without the need to wear a headset. Users commented that with AR, they were able to see and interact with VR users. The enhanced visual cues from the mirrored display was a good facilitation for communication.
These aspects that we have reported here account for factors of communication that are important to the design of hybrid reality environments. We believe that user preferences for full immersion or for augmented reality can be diverse in the population, and that such designs are important for the wide adoption of VR and AR for social use.

Discussion
This research investigated how communication differs between hybrid VR and AR environments as indicated by perceived spatial presence, copresence, and social presence. In this section, we discuss our results and findings in view of the questions asked.

Visual and Spatial Information and Shared Virtual Space
Are there perceived differences for perceived presence between HVAR and SVR (RQ1)? Our findings indicated that greater spatial presence, copresence, and social presence were experienced in SVR compared with HVAR (H1). Based on the results, we can confirm that rich visual and spatial information in VR contributed to the increased perception of spatial presence compared with AR. VR users were immersed in a simulated environment with rich interactivity whereas AR users were subject to distractions from the physical environment. VR also provided the physical context where objects were placed. The spatial information was mapped to the embodied experience making use of participants' physical body in both navigation and interaction. Within HVAR, AR users had fewer interactions to explore compared with VR users due to the lack of spatial information presented in the application: they could only see the objects on the cube but not in a virtual environment. Previous studies have shown that rich interactivity and the exploratory behavior of VR users tended to increase the sense of believability (Ch'ng, Li, Cai, & Leow, 2020). The comparison of users in VR and AR found greater spatial presence for SVR than for HVAR. In addition, the higher spatial presence perceived by the VR users for SVR than for HVAR indicated that the increased spatial presence felt by a user could contribute with the other users' sense of presence in the shared virtual space.
Secondly, we observed that a shared virtual space with the same amount of visual and spatial information in SVR contributed to both higher copresence and social presence. This was expected as users shared symmetric interactions and the same amount of visual and spatial information in SVR. A shared virtual space is helpful in supporting mutual awareness and thus connections were easily established. Our study confirmed Grandi et al.'s (2019) findings that perceived social presence is greater in an environment with symmetric interactions (SVR) compared with environments with asymmetric interactions. We also confirmed that perceived spatial presence and copresence were greater in SVR due to the shared visual and spatial information. In summary, the rich visual and spatial information in VR led to greater spatial presence compared with AR; the shared virtual space with the same amount of visual and spatial information in SVR contributed to higher perceived copresence and social presence compared with HVAR with asymmetric interactions. Such perception of a shared space accounted for our observed user activities following social norms and mannerisms, such as keeping objects in order and greeting each other.

Visual Cues of User Interactions
Are there perceived differences in presence between VR and AR users within HVAR (RQ2)? We found that users in VR did perceive greater spatial presence compared with users in AR (H2a). However, there were no significant differences in perceived copresence (H2b, rejected); also, contrary to our expectation, users in AR perceived greater social presence compared with users in VR (H2c, rejected). Our initial hypothesis statement was based on the fact that users in VR had more control over virtual objects, around which communication was expected to occur. We evaluated our observations and interviews and found that the phenomenon was associated with visual cues of user interactions.
AR users in the co-located sessions were able to see in real time, visualization of VR users' interactions through the mirrored display. Despite the fact that the 40-inch display was non-immersive, it allowed AR users to see the paired partner's interactions. This made the intended actions of the VR users transparent through the mirrored display, which provided AR users with more visual cues. We believe this contributed to their increased sense of social presence. In addition, it is reasonable to speculate that AR users felt a greater sense of social presence because of the cues they obtained from the co-located setting where they could see and talk to the VR user. On the other hand, the only cues that VR users had of AR users' interactions were via the object rotations and the linked spatial audio. Such cues were limited, although they did inform VR users of interactions from AR users. In using virtual objects as the interface through which VR and AR users connect, we can begin to understand that communication requires users to be represented by avatars. VR users knew through the rotation of objects that another user was in the shared space, but that someone was not represented in the simulated view. This affected VR users' perceived social presence in HVAR.
Our study also demonstrated that users showed no significant differences in perceived social presence between using AR in HVAR and using VR in SVR. This group of users were able to see the paired partner's interactions in both sessions, either via mirrored display or embodied in virtual avatars. These visual cues for user interactions were important in facilitating perceived social presence. We conclude that the visual cues of user interactions such as the mirrored display in HVAR and the embodied avatars in SVR, can greatly contribute to the perceived presence and as such facilitate user communication in both environments.

Shared Activity Time Ratio as an Indicator of Social Presence
Does shared activity time correlate with perceived social presence (RQ3)? Our research found positive correlations between shared activity time ratio and social presence in both HVAR and SVR (H3). Users who spent a greater ratio of time in shared activities also reported greater social presence. These findings can inform future research in communication mediated by immersive technologies, and make use of user activity data in the analysis of social presence. The shared activity time ratio can be used to cross-validate the results of the self-reported measures. If self-reported measures are not feasible, such as for studies of public exhibitions in-thewild, the analysis of shared activity time ratio from the user activity data can be used to gauge users' social presence. We believe that the monitoring of user activity at system runtime and the analysis of time spent in shared activities can be an effective indicator of social presence for a collaborative environment.

The Sense of Social Distance in HVAR and SVR
In addition to our proposed research questions, we further investigated the factor of acquaintance on perceived presence. Acquainted pairs perceived significantly greater spatial presence, copresence, and social presence than unacquainted pairs in SVR. However, no significant differences were found in HVAR. Users reported increased intimacy, involvement, and immediacy in SVR, indicated by the greater perceived others' copresence in SVR than in HVAR. Sharing a virtual space in SVR allowed the perception of users and their proximity through virtual avatars. Although we observed higher counts of interactions between virtual avatars in acquainted pairs, unacquainted pairs tended to have less interactions in SVR. Comments received in the interviews showed that unacquainted pairs tended to keep a distance and prevented themselves from intruding into another's activities, much like one would do in a public space. On the other hand, since VR and AR users in HVAR were situated in two different worlds with differing realities in the spectrum, and that communication was via virtual objects, users were less likely to be aware of spatial proximity of their partners. In such cases, acquaintance was not an influencing factor for perceived presence. In the interview, acquainted pairs reported that they expected more interactions from partners, with demands to be able to see the partner who is using AR. However, unacquainted pairs commented that HVAR's limited access to interactions of users shifted their attention to their own experience, without having to provide reactions to others. We suggest that, in HVAR, the sense of social distance caused by the lack of avatar representations was a departure from the natural connection that individuals are used to at spatial proximity. As a result, this effect made HVAR more acceptable and more comfortable than sharing a virtual space in SVR for unacquainted pairs.

The Spectator Experience with AR
Here, we extend the results of our work and conceptualize its application to the spectator experience to conclude our work. Reflecting on our statistical results and interview feedback, we argue that AR can be used for including audiences in scenarios that support the spectator experience and for complementing and enriching VR in social contexts. Reeves, Benford, O'Malley, and Fraser (2005) introduced the idea of designing the spectator experience in public spaces. They conceptualized the approaches for designing the spectator experience based on manipulations and effects. It is often the case that spectators are able to see a VR user's interactions via a display. However, it is difficult for them to experience what they can see without them being in the space themselves. Our development and understanding of the HVAR experience can be extended for the spectator experience-by bringing spectators into a hybrid space where the VR user becomes the performer, and the AR users then become active spectators. In this case, a single set of VR equipment can be used together with multiple, more accessible mobile devices. This will mitigate the isolation of VR users and benefit users who prefer not to wear an HMD. Our observations of the spontaneous cooperation on objects and the "guided tour" that users initiated revealed to us how the asymmetric interactions for HVAR environments can be leveraged to facilitate the future of communication.
The concept of HVAR for communication can engage bystanders into the experience in public spaces, and also in private spaces that involve families and friends. The performer-spectators relationship may be inverted and extended to the Teaching and Learning environment, where student interactions are monitored in VR and teachers manipulate elements in AR. Such use of HVAR may provide a safe environment to ensure student safety when using VR. Future use of HVAR should consider how perceived social presence in VR can be enhanced via the use of visual, aural and spatial cues. Previous research has shown that the use of virtual avatars in VR, even with a simple animated guide (Li, Tennent, & Cobb, 2019), can increase the sense of social presence. Future work may investigate whether augmenting an avatar around observed virtual objects can help facilitate users' perception of social presence and support communication.

Conclusion
In this article, we investigated the effects of presence and its relation to communication in Hybrid VR and AR environment (HVAR) compared to Shared VR environment (SVR). We detailed the design and implementation of our HVAR environment that supports synchronous and co-located sessions around virtual objects. We conducted a robust set of experiments with 52 participants in 26 pairs using both HVAR and SVR. Our results compared between HVAR and SVR confirmed our hypotheses in terms of reported spatial presence, copresence, and social presence. We further demonstrated that the shared activity time ratio is an effective indicator of social presence for a collaborative environment.
At the beginning of the article, we asked how communication differs between hybrid VR and AR environments. We found that, overall, the complete simulated visual and spatial information in VR contributed to greater spatial presence than AR. VR users also perceived greater copresence and social presence within the shared virtual space for SVR than for HVAR. Despite the differences, visual cues from the mirrored display in HVAR and from embodied avatars in SVR have significant effects in influencing perceived social presence and as such facilitate communication. Another observation was that the lack of avatar representations caused an increase in the sense of social distances in HVAR. While this may be seen as a negative effect, it was actually more acceptable and comfortable for unacquainted pairs compared with the sharing of a virtual space in SVR. We demonstrate that AR can be used for including audiences in scenarios that support the spectator experience and for complementing and enriching VR in social contexts. Our design and evaluations of HVAR can inform the future design of multi-device social environments that support hybrid realities. Our results and findings contribute to extending knowledge in the understanding of how presence affects communication in hybrid VR and AR environments. Future research in hybrid VR and AR environments will investigate the use of avatar representations and the effects of virtual proximity on social presence and communication.

Limitations and Future Work
Our current study has some limitations. Here, we identify some improvements for future HVAR research. First, the sample of our study reflected communication between university students and staff. Most of the pairs signed up to the study as acquainted pairs and the sample for unacquainted pairs was limited. Research to explore a larger sample of users with more complex interpersonal relationships will be needed. Addition-ally, whilst the co-located HVAR experience did provide users in AR with a more comprehensive view and awareness of the social context, it made users relatively passively engaged in the VR environment. The primary concern for the future design of HVAR will be the use of available visual and spatial information to provide accessible cues for interactions, and to increase the perceived social presence in VR.