Facebook is preparing a virtual keyboard for any surface

Researchers from Facebook have just presented a virtual keyboard based on hand tracking thanks to the cameras integrated into a virtual reality headset. It does not require any additional devices, and can be used on any flat surface.

This will also interest you

The Oculus Quest and Quest 2 virtual reality headsets, which have just been released, offer an experimental hand tracking function that is very convenient for manipulating certain objects or navigating menus without a controller. Using the external cameras, the helmet is able to identify the hands and track the movement.movement. However, the gestures are limited, and the follow-up a little capricious.

Obviously FacebookFacebook, of which OculusOculus is a subsidiary, is still working on this function. In a paper presented at the UIST 2020 conference, researchers from Facebook Reality Labs unveiled their work on a new hand tracking technique that is accurate enough to write on a virtual keyboard.

The use of a virtual keyboard requires for the moment to wear gloves with markers. © ACM SIGCHI

Typing as fast as on a physical keyboard

The technique makes it possible to write on a flat surface, without a physical keyboard or touch input. The system uses the cameras of a helmet and a temporal convolutional neural network to represent the movement of the hands. It integrates a language model and a beam search algorithm to compensate for tracking errors, but also hand placement errors. Without the tactile sensation of the keys, users’ hands tend to drift slightly. To improve finger detection, users had to wear gloves with markers.

The results are very promising, with a typing speed of 73 words per minute, equivalent to that obtained with a physical keyboard. However, the researchers still have work to do, because the virtual keyboard is currently limited to lowercase letters and works poorly with any word that is not in the dictionary. In addition, the system had to be calibrated for each user, requiring each time to record more than one hour of input on a physical keyboard. Their tracking system is also much more accurate than that present on helmets like the Quest. However, this technology could eventually make it possible to use virtual or augmented reality headsets to work anywhere.