helmets keep a huge margin for progress

A prototype of Sony’s PlayStation VR at the E3 show in Los Angeles, in 2015. MARK RALSTON / AFP

Engineers have been working since the 1960s to realize the fantasy of a headset completely immersing its user in a virtual world. But deceiving our eyes and our body is delicate and, over the decades, many attempts have failed. The new generation headsets that appeared in 2016 are the first consumer products to offer a tolerable experience and, in five years, the models from HTC, Oculus, Sony and Valve have progressed, but they have only covered a small part of the necessary path before reaching, perhaps one day, a sensation close to reality.

First challenge: widen the field of vision. Mainstream models give the impression of looking at the image with blinkers as their field of view (between 90 and 110 degrees) is narrow. To reproduce a natural field of view, you would have to go up to 240 degrees. “It will probably be necessary to use curved screens to avoid excessively weighing down headsets with huge screens and bulky lenses,” Graham Wheeler, CEO of the HTC Vive headset manufacturer, tells Le Monde. Moreover, we could also talk about the intensity of the brightness of the screens, still a hundred times lower than that of natural light in the middle of summer (120,000 lux).

Even if the resolution of their small screens has doubled in five years, it is still very insufficient

The other big challenge is to display perfectly sharp images. Our televisions and our smartphones are already capable of this, at the distance where we look at them, but the headsets are still far from it. Even if the resolution of their small screens (one in front of each eye) has doubled in five years, to reach about four million pixels, it is still very insufficient and we can still guess the grid of small luminous cubes that line their screens. The images seem blurred, the texts lack legibility.

For Mr. Wheeler, “it will be necessary at least to reach a resolution of 16 K per eye if we want to allow the small print of a soda can to be read”, to understand that it would be necessary to stack 132 million pixels on screens seven centimeters wide — a challenge. The 8 K TVs, incomparably larger, cap at 32 million pixels.

A patent for curved screen glasses filed by Samsung in 2019. SAMSUNG

Still, in the case of video games, it would be necessary to calculate such detailed images that even a PC gamer could not provide. Unless the headsets adopt a trick that took its first steps in 2019: foveated rendering. The headsets using this technology follow the movements of the eye to detect the area they are looking at and display maximum graphic quality in that precise place, while degrading the quality on the sides, where we see less precisely. The advantage is to reduce the computing power requirements.

The other solution would be to calculate the images away from the headset, on super-powerful remote computers, then route them to the headset, either by fiber (which homes are still very variously equipped with) or by 5G. But the 5G mobile phone network is not ready for this use either: its reaction time is still, and for a few years, much too slow.

• Offer natural orders

The controllers track the movements of the hand rather correctly and make it possible to grasp an object in a relatively natural way by tightening the fingers on their handle. But the precise movements of each finger are just beginning to be tracked, especially by the First Index Finger, such as the joints (wrists, elbows, shoulders, knees, pelvis) of the human body. Several ways are envisaged to achieve the monitoring of the movements of the whole body: the first, binding, is to put on a suit lined with detectors; the second resorts to cameras which monitor the movements of the limbs in real time. “We will probably need to have several external cameras work together. The ones we have integrated into the helmet cannot see all the parts of the body at the same time,” believes Graham Wheeler.

The hand must meet a variable resistance from one object to another

To offer a perfectly natural interaction, touch must also be taken into account. The hand must encounter a variable resistance from one object to another — it is not the same when holding a knife or a foam ball. The body, on the other hand, is able to perceive all kinds of contacts and feels resistance when an obstacle prevents it from moving freely. “This is the problem to which we imagine the most difficult to find a solution, regrets Mr. Wheeler. The tracks that we have allow us to start considering stimulating the touch by ultrasound, in order to give a sense of the click. But this is only a very small beginning of the answer to this very complex problem. »

The accessory positioned below this helmet is able to read the position of the lips. It was released by HTC in March 2021. HTC

The face is a separate case, on which the helmets are progressing rapidly. “We are now able to track the movements of the eyes, which by extension gives us the position of the eyebrows. We can also follow the movements of the mouth with the HTC device, which gives us an idea of the expressions of the middle of the face” testifies Graham Wheeler. A track that Oculus is also pursuing, according to its main shareholder Mark Zuckerberg. This new information will enrich the expressiveness and social interactions of virtual avatars.

• Freeing up travel

Autonomous headsets like the Oculus Quest offer the possibility of evolving freely, without a cable limiting our movements. In order to avoid impacts with furniture or walls, some helmets now offer the possibility of defining a safe evolution zone. When walking outside this area, the virtual reality pauses and the headset broadcasts an image of the environment that surrounds it, a bit as if it became transparent. Practical, this function does not correct the fundamental problem: unless you have a shed for the game, it remains impossible to walk for a few meters without being interrupted.

Even if many applications or games are practiced sitting down, the feeling of moving naturally is, for the moment, unattainable. To this problem, we can only consider, for the moment, bulky and rudimentary solutions, such as the strange Omnione mat on which we can move by sliding our feet, or very expensive, such as the omnidirectional electric treadmills imagined by NASA or by Infinadeck.

The problem is all the more annoying because it feeds another flaw in VR: its tendency to make you nauseous. To avoid it, it is necessary that the movements displayed in the headset correspond perfectly to those of the body, otherwise the inner ear will be disoriented. “VR must be modeled on our senses, explains Balthazar Auxiètre, the creative director of the Fisherman’s Tale game. We cannot exceed a certain threshold of unnaturalness. “At the moment, the main solution is to drastically limit the player’s movements. Even if omnidirectional treadmills became widespread, how, for example, to reproduce the shocks received by the body in a game where a kayak occasionally abuts against the bank?

Reactivity too, if the image does not immediately follow the movements, can cause stomach upset. “In 2016, the first helmets already offered sufficient reactivity. We have managed to maintain it for five years, despite the increase in the resolution of the images. But the developers who design the VR experiences sometimes make mistakes “that can cause significant delays.

Many helmets weigh half a kilo. To increase the comfort of virtual reality sessions, they will have to lighten up considerably. To improve their social acceptability, they would also benefit from being more discreet. “The vision we have at HTC is that of contoured glasses, like those we use when skiing and which weigh less than a hundred grams. “We are far from it, especially if these helmets must incorporate a battery to avoid any connection.

A prototype of virtual reality glasses unveiled by Panasonic in 2021. PANASONIC

Giving a perfect illusion of reality is a distant goal, probably unattainable for VR headsets. But they will be able to make significant progress by at least learning to deceive better our eyes, with a wider and sharper vision, and by following the movements of our body and our hands more precisely.

Read also Five years after the revival of virtual reality headsets, where are the games at?

Nicholas Six