By Gustavo Melgarejo

Virtual reality (VR) combines mechanical and electronic components in a three-dimensional computer environment to create artificial interactive environments that users feel are real.

This technology has become popular in recent years, mainly because of its ability to simulate situations or environments that, in the real world, could put people at risk. For example, VR has been widely used in the treatment of phobias, where patients can be subjected to situations of acrophobia (fear of heights) or arachnophobia (fear of arachnids) without actually compromising their integrity and maintaining control at all times.

There are many applications for VR. Beyond the treatment of phobias, this technology has had a great impact in areas such as sports training, education, vehicle simulation, medical devices, and even motor and cognitive rehabilitation systems. Regardless of the application, one of the most interesting questions related to the development and study of VR is how users perceive these virtual simulations as “real”.

Presence and Immersion in Virtual Reality

In order to understand how a user perceives a VR experience as “real,” it’s important to know and differentiate between two terms: immersion and presence.

Immersion is associated with the technology available to a VR system; in other words, how many different types of technology are available to stimulate the various sensory senses and create vivid experiences.  This includes advanced VR goggles, as well as “cave” systems that consist of a room equipped with high-definition projectors to provide an immersive sensation. In addition to this, there are stereo sound systems, which are high-definition multi-channel systems that, in conjunction with special controls and haptic devices (those that generate tactile responses), generate stimuli similar to those found in real-world scenarios.

On the other hand, presence is a person’s physical and mental sense, which makes them feel like they are in a place even though they are not there. This sense is basically a result of the immersion of the system. As long as the system can provide stimuli that deceive the sensory senses, a person can forget that they are in a room using a VR viewer and begin to experience it as if they were really there. Therefore, when it comes to assessing a user experience or a level at which the user perceives a virtual simulation as real, a sense of presence will be used (see Figure 1).

How is a sense of presence assessed?

Initially, sense-of-presence studies began in the 1990s based on questionnaires where users answered questions related to their experience. However, over the time, several problems were detected with the use of these questionnaires since in general they were wholly subjective and there was no standard for writing the questions or answering them. In addition, when answering this type of questionnaire, the user may be influenced by personal and cultural factors, not to mention the time elapsed between the virtual experience and the completion of the questionnaire that may distort “recall,” further compromising the results.

For this reason, exploration began of the so-called Affective Computing Systems (ACS), which promised a more novel and accurate approach capable of providing both qualitative and quantitative results. Specifically, an ACS makes it possible to analyze the responses of the autonomic nervous system, which, in turn, are reflected through physiological signals from the human body, providing instantaneous and involuntary responses related to the emotional state. Among the most popular physiological signals are the electrocardiograph (electrical activity of the heart), the electromyograph (electrical activity of the muscles), electrodermal activity (sweating changes in the skin), and the electroencephalograph (electrical activity of the brain). After obtaining these signals, they are analyzed to determine certain characteristics, such as heart rate, sweating level, a person’s state of alertness, etc. Later, an interpretation is made to determine the emotional state of the subject, making it possible to tell, among other things, if they are stressed, happy, sad, or angry.

But how does the subject’s emotional state relate to their sense of presence? According to empirical evidence obtained from numerous studies over the years, if a VR experience can evoke a specific emotion in a user, just as it would in the real world in the same scenario, then the subject is said to be experiencing a sense of presence. Revisiting the example of phobia treatments, the scenarios shown often generate anxiety and stress in test subjects, just as they do in the same real-world situation. Figure 2 summarizes the above process, which has generated great advances in the study of VR.

VR advances and outcomes at Tecnológico de Monterrey

In particular, the team at Tecnológico de Monterrey has focused on analyzing new strategies to design more efficient VR systems to evoke a sense of presence. This also includes new approaches to a sense-of-presence analysis to provide metrics that provide us with more concrete study outcomes.

Between 2015 and 2019, a VR-based bicycle simulator was developed, which allowed us to have experiences similar to those in the real world. For this purpose, mechanical, electronic, and control systems were integrated to provide feedback to users through the pedals and the handlebars, thus achieving an immersive experience with limited technological resources. It became evident that the problem with using questionnaires was that some users weren’t sure how to answer the questions once they had completed their virtual tour on the bicycle simulator.

Subsequently, between 2018 and 2021, a theoretical framework was developed using proposed metrics called “Error of Presence” and “Sense of Presence degree”. These metrics made it possible to address the problem of analyzing presence in VR systems with mathematical models that would provide more precision in their analysis and develop a control strategy to change the VR content according to the presence status of the test subjects. The proposed methodology was validated with experiments, and it was partially demonstrated that a sense of presence can be controlled by generating specific emotions in users while using a VR system. Including mathematical strategies in the sense-of-presence study set a precedent for this state-of-the-art technology. It is worth mentioning that these studies and their outcomes involved the important collaboration of colleagues from the National Polytechnic Institute (UPIITA) and CINVESTAV Guadalajara Unit.

Even with the outcomes obtained, there is still much work to be done since physiological signal processing and emotion detection remain a challenge given the complexity of the human body and the wide variety of reactions a person can have.

Undoubtedly, this is an area with a promising future because the evolution of affective computing systems and their integration with VR will make it possible to create personalized experiences. For instance, let’s think about the possibility of a video game that adapts if you are very stressed or becomes more challenging if you are getting bored. Another possibility could be physical rehabilitation routines that adjust to the individual progress being made. All of this could create a real paradigm shift in this area.

If you would like to know more, check out the following articles:

G. Hernández-Melgarejo, A. Luviano-Juárez and R. Q. Fuentes-Aguilar, “A Framework to Model and Control the State of Presence in Virtual Reality Systems,” in IEEE Transactions on Affective Computing, 2022. doi: doi: https://doi.org/10.1109/TAFFC.2022.3195697

G. Hernández–Melgarejo, D. Flores–Hernández, A. Luviano–Juárez, L. Castañeda, I. Chairez, and S. D. Gennaro, in ISA Transactions, vol. 97, pp. 336–351, 2020. doi: https://doi.org/10.1016/j.isatra.2019.08.002

Slater M, Sanchez-Vives M, “Enhancing our lives with immersive virtual reality,” in Frontiers in Robotics and AI, published online 19 December 2016. doi: https://doi.org/10.3389/frobt.2016.00074

Marín-Morales, J.; Llinares, C.; Guixeres, J.; Alcañiz, M., “Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing,” in Sensors, 2020, 20, 5163. https://doi.org/10.3390/s20185163

Author

Gustavo Hernández Melgarejo, [email protected]

I am a Mechatronics Engineer with a Ph.D. in science specializing in electrical engineering. I am currently a PostDoc researcher at the Institute of Advanced Materials for Sustainable Manufacturing at Tecnológico de Monterrey. My research focuses on affective computing systems, and the implementation of virtual reality systems, as well as the design, instrumentation, and control of mechatronic systems. https://orcid.org/0000-0003-1430-7567