The next time you’re scrolling your phone, take a moment to appreciate the feat: The seemingly mundane act is possible thanks to the coordination of 34 muscles, 27 joints, and over 100 tendons and ligaments in your hand. Indeed, our hands are the most nimble parts of our bodies. Mimicking their many nuanced gestures has been a longstanding challenge in robotics and virtual reality.

There are several ways to record those motions so that robots can replicate them in real time, but each approach has drawbacks, reports New Atlas.

Until visual obstructions get in the way, cameras are fairly good at capturing a wide range of motions. Wearing sensor gloves restricts the human hand's normal movement and sensation, yet they can convey detailed motion data and are unaffected by barriers. In order to forecast hand movements, another technique measures electrical impulses from muscles using sensors on the wrist or forearm. However, these sensors are not very good at picking up on small in-between motions and are susceptible to background "noise."

Using ultrasonic imaging, the new method created by MIT researchers is the most accurate and dependable to date. Small ultrasound stickers, about the size of a watch, paired with compact electronics, are placed on the wrist in a wristband. As the fingers move, this configuration produces crisp, continuous images of the tendons and muscles.

To explain how it works, Gengxi Lu, one of the researchers, uses the analogy of puppet strings.

Related Device Uses AI to Read Gestures Even During Intense Activity

"The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers," he says. "So the idea is: each time you take a picture of the state of the strings, you’ll know the state of the hand."

The ultrasound scans show each of the 22 degrees of freedom that each finger possesses. At first, researchers attempted to match those movements with the pictures, but this proved to be too difficult for people to perform in real time. Rather, they trained AI to identify patterns in the ultrasound pictures and forecast hand motions, and it was flawless!

In order to evaluate this system, volunteers performed all 26 American Sign Language letters and interacted with various objects, including a tennis ball, pencil, and scissors. The wristband was able to correctly predict hand positions in every instance.
Additionally, the bracelet was evaluated by the researchers as a wireless controller for a robotic hand that could replicate movements in real time, including playing a basic piano melody.

The ultimate goal is a wearable, compact hand tracker that anyone can use to remotely control virtual or robotic items. The AI may potentially be trained to perform a variety of activities, including assisting with surgical procedures, interacting with virtual reality environments, and operating equipment without touch, by gathering a large amount of hand movement data.

The research was published in the journal Nature Electronics.