New first-of-its-kind wearable technology from researchers at Cornell University and the Korea Advanced Institute of Science and Technology equips off-the-shelf smartwatches with AI-powered micro sonar capable of tracking hand movements, which could support assistive technologies for users with limited mobility or speech and be used as a controller in augmented reality and virtual reality environments.

The system, called WatchHand, has the potential to revolutionize how we interact with our devices by continuously tracking hand poses in real time using smartwatches and their built-in speaker and microphone, the researchers said.

It is the first time AI-powered acoustic sensing for hand-pose tracking has been implemented on off-the-shelf smartwatches without the need for additional hardware, reports EurekaAlert.

“In the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch,” said Chi-Jung Lee, a doctoral student in the field of information science. Lee is co-lead author of the paper, which will be presented at the Association for Computing Machinery (ACM) CHI conference on Human Factors in Computing Systems.

Related Amazfit’s T-Rex Ultra 2 Features Biotracking, 30-day Battery

Existing wearable hand-tracking prototypes require bulky hardware, rendering them impractical for everyday use, the researchers said, but WatchHand uses the existing microphone and speaker within standard smartwatches. Equipped with WatchHand, the smartwatch’s speaker emits inaudible sound waves that bounce off the hand and back into the watch’s microphone, creating an echo profile image. WatchHand’s machine learning algorithm, which runs on the smartwatch, reads this echo profile and estimates the hand pose in 3D and in real time.

All hand-pose data and processing would take place locally on the watch, meaning that personal data wouldn’t be shared, the researchers emphasized.

WatchHand was tested with 40 participants across four studies, totaling around 36 hours of gesture data. It was evaluated across different smartwatch models, on right and left hands, and in noisy conditions, and was found to reliably track finger movement and wrist rotations.

Its performance isn’t perfect, researchers noted. For starters, it works on Android smartwatches, not Apple iOS. While performing well in noisy spaces, WatchHand had trouble registering hand poses if the user was walking, for instance.