5 lookout points to observe UX in virtual reality
When we first ran VR testing, we were in a phase of great exploration; we gave tasks to users and meanwhile, observers and facilitators were taking notes, based on a list of pre-set goals for each testing. After conducting several tests on different applications, we clustered the points of attention and defined 5 focal points to structure the test when planning, executing, observing and analyzing. The rating we gave to the different areas helps to indicate the user experience (UX) offered throughout the application, based on the usability hierarchy of needs, going from functional to meaningful. To obtain user engagement, we should aim to go beyond a pure functional experience and transmit joy, trust and reduce emotional strain. The more meaningful an experience becomes, the more memorable it will be and the more likely it will draw the user back for more. Emotional design and true eye for detail will change how our users feel. He or she should be the protagonist of the story, having the power to make key decisions and experience the consequences of those decisions through the VR adventure.
We will go into more detail on the gap we identified between normal and VR usability testing within the 5 areas of focus.
Effectiveness: Assessing the skills of the surgeon after his or her VR training
Was the story clear? Has the user looked around and seen all the objects in three dimensions in the different areas of the application? Did the virtual tour look like reality? In short, did the VR experience fulfill its value proposition?
VR applications often offer a non-linear experience, which makes it trickier to evaluate success. The user is the driver of the experience, he or she decides which path to take and the facilitator shouldn’t lead the way.
Directness and time of completion do not necessarily need to be the main success factors in VR applications. Often the path to fulfill a goal is just as important as the destination itself. Test hypothesis and the means of success should therefore be carefully defined, including the different stimuli that define the experience. To assess its effectiveness, we usually put a lot of emphasis on the post test and post task interview we have with the users in which we aim to understand what the user thinks he or she has learned, seen, felt or heard. To analyze the user’s behavior through the complete journey and guarantee no details have been missed out, we record the testing and have observers watching the users from another room.
VR applications are great to make users do and learn what they can’t in real life. To guarantee users get the story right, they shouldn’t miss out on the essential. Even big, flashy signs are overlooked if they are not placed in the direction the user is looking.
If we relate this to the content zones as defined by Google interaction designer Mike Alger; essential elements that are part of the core experience should be placed in the main content zone. Clear guidance and signs should be given in case the user is supposed to look into other directions. Let’s imagine an application that offers a virtual walking tour through a museum with different rooms. The user should be encouraged, through various stimuli, to turn 180° and take a look at what is happening behind him or her. This could be done, for example, by using visual elements such as lights or sounds that come from a clear direction to attract his or her attention.
Interpretability: Evaluating how much sense outer space makes for the astronaut
Did users experience sounds, environment, lights and so on as they were part of a three-dimensional space? Were movements in the physical reality reflected correctly to the virtual reality? Did the environment makes sense at all?
The first reaction should be captured to observe the user perception of the virtual world. It’s necessary to let the user free ride for some time in the environment without giving him specific tasks. While testing, user behavior and comments should be carefully observed to understand how he or she interprets the environment and objects. In one test we carried out, a user was interacting with a simple application which displayed a table with different tools on it. The user picked up one of the tools and accidentally dropped it. The tool simply floated in the air. The user’s comment was pretty obvious: “This 0 gravity doesn’t make any sense, I’m in a workshop, not on the moon”.
When interacting with VR applications, users form part of a virtual world where the impossible becomes possible: flying, walking on the moon, having super powers and much more. To give sense to this world, there should be clear rules about what is possible and what is not, to gain the trust of our users. Designing a complete new world with no reference to the real world is very challenging. VR apps should be built on what the user knows in the real world and be an extension of that. Consistency with the physical world helps guide users through the virtual one.
Comprehension: Testing if superman masters his own powers in the VR zone
Have users understood how to complete a simulated action? Are they able to physically interact with elements using body movements? Are the different features of the application clear? Essentially, we measure whether the global idea and journey have been understood .
We encountered quite a different success rate between experienced users who had already interacted with VR applications prior to the test and non-experienced users, who took more time getting used to the environment and interacting with the controllers. As shown in the graph below, this difference was mainly visible at the beginning of each test, but became less notable towards the end. In other words, we didn’t see a clear difference in the steepness of the learning curve between the two types of users.
Therefore, when setting the sample, the users’ previous experience should be taken into account and included when analyzing the success rate of the initial tasks. It might be difficult to interact with the user during the test or to ask him or her to think out loud because of the presence of extra stimuli such as sounds. We aim to ask as little questions as possible to not to break the virtual flow experience. Of course, sometimes it is inevitable to interrupt the user after completing a task to gain insights into his or her motivations.
If users don’t understand an application, they won’t use it. VR allows to offer multi-sense feedback, storytelling, guidance, reference points and a step by step introduction of new elements which increases the comprehension rate significantly. Think about a VR application that teaches mechanics to assemble a car: the cognitive load can be minimized by keeping the workspace centralized in the field of focus, correcting him with haptic feedback such as friction through the gloves he is wearing while assisting him with a voice guide introducing new components step by step.
Satisfaction: Examining the joy of the virtual rollercoaster ride
Did users like the application? Did they feel safe? Was the experience enjoyable? Does it seem worth the wait to come home and interact with it? We want to understand if users would actually use the application in a non-test environment.
When interacting with a VR app, part of the user’s face is covered which makes the observations of the users’ spontaneous reactions less direct. As mentioned before, we want to interrupt the experience as little as possible.. We aim to understand the level of satisfaction during the post test interview in which we ask the user to describe his or her experience with questions such as “Which adjectives would you use to describe the experience ?” “How did you feel in that specific moment?” We combine this with a post-test survey where we give different statements around the intention of use and ask the user to rate those from 0 being completely disagree to 5 being completely agree.
We should focus on creating a core experience that is enjoyable, challenging and rewarding as well as implemented. Imagine a wonderful designed environment of the moon in a virtual app. Users will be amazed and take time to look around, but if the core experience, for example fighting moon monsters, doesn’t fulfill its value proposition, they won’t be engaged for more than a few minutes.
Comfort: Testing ergonomics and sickness within the twilight zone
Was the experience the same for left-handed and right- handed users, for users who wear glasses and those who don’t ? Did users get motion sickness after interacting with the application? Were users more comfortable seated or standing up ? Was the volume of the background sound perceived as comfortable?
When evaluating the comfort of a VR application, we have to consider both physical comfort and non physical comfort.
Physical comfort includes everything related to the body movements of the user; the way he or she uses the controllers, moves his head left, right, up and down, walks around, the volume of voices and background sounds, the match between what the user sees and what his or her body is used to, etc. In order to carry out a proper evaluation, it is important to simulate the same conditions the user would experience in a normal situation and not in a test situation. VR glasses should be comfortable and the test room should be safe, without any physical obstacles. Also, the time that the user is wearing the VR glasses should be respected and similar to a realistic use. When setting the test sample, anthropometric factors should be included, to ensure the full range of potential users is covered.
Non physical comfort covers how comfortable a user feels within the virtual world. Sizes of objects, distances to items and readability of texts for example define the ergonomics within the virtual world. The user should onboard, live and leave the experience in a conscious way.
VR offers users the chance to do the impossible, but should not let users do the unwanted; especially when an app has open access. The user should be protected both in the real and virtual worlds. Apps should be designed in a way that respect users’ norms and values. For this, part of the testing should include studying their values and norms to make sure the app is aligned with them. Users should be told that they can leave at any time during the test if they feel uncomfortable.
When users are interacting with VR, rather than with other systems, their perception and acceptance can be influenced by culture, background, age, sex and identity. They form part of the VR world and this world need to fit both their mental modal and stature. Inclusive design is fundamental to find a metaphor that works for all targeted users.
Long live usability testing!
VR technology is expanding at a high speed and UX should co-lead this revolution. Unlike in the mobile and desktop world, there are still no strict rules that define how VR UX should be outlined. The best way to find out what works and what doesn’t is to reach out and test solutions with end users, as early as possible in the design process. We always
welcome as many stakeholders as possible, such as developers, product owners, business managers, 3D designers, interaction designers, etc. to our observation room to take part in the design process and to make sure that learnings are adopted.
At our research center, we are first line fans of usability testing. Users mean the world to us, they inspire us, give us the most valuable insights and help us make design decisions in the most effective way.
Viva UX research, creating better design opportunities every day!