A Smartwatch is a popular device that helps improve its wearer’s health. Now University of Washington researchers have developed a smartwatch app for deaf and hard-of-hearing people who want to be aware of nearby sounds. Dubbed SoundWatch, the wearable device picks up a sound the user is interested in — examples include a siren, a microwave beeping or a bird chirping — SoundWatch will identify it and send the user a friendly buzz along with information about the sound.
The researchers presented their findings at the ACM conference on computing and accessibility.
Read more: This Smart Glove Interprets Sign Language In Real Time
“This technology provides people with a way to experience sounds that require an action — such as getting food from the microwave when it beeps. But these devices can also enhance people’s experiences and help them feel more connected to the world,” said lead author Dhruv Jain, a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering. “I use the watch prototype to notice birds chirping and waterfall sounds when I am hiking. It makes me feel present in nature. My hope is that other d/Deaf and hard-of-hearing people who are interested in sounds will also find SoundWatch helpful.”
The team started this project by designing a system for d/Deaf and hard-of-hearing people who wanted to be able to know what was going on around their homes, reports UW News.
“I used to sleep through the fire alarm,” said Jain, who was born hard of hearing.
The first system, called HomeSound, uses Microsoft Surface tablets scattered throughout the home which act like a network of interconnected displays. Each display provides a basic floor plan of the house and alerts a user to a sound and its source. The displays also show the sound’s waveforms, to help users identify the sound, and store a history of all the sounds a user might have missed when they were not home.
Because smartwatches have limited storage and processing abilities, the team needed a system that didn’t eat the watch’s battery and was also fast and accurate. First the researchers compared a compressed version of the HomeSound classifier against three other available sound classifiers. The HomeSound variant was the most accurate, but also the slowest.
To speed up the system, the team has the watch send the sound to a device with more processing power — the user’s phone — for classification. Having a phone classify sounds and send the results back to the watch not only saves time but also maintains the user’s privacy because sounds are only transferred between the user’s own devices.
The team tested the SoundWatch app in March 2020 — before Washington’s stay-at-home order — with eight d/Deaf and hard-of-hearing participants in the Seattle area.
Read more: Buzz: A Wearable Wristband That Allows You to Sense Sound Through Your Skin
People found the app was useful for letting them know if there was something that they should pay attention to. For example: that they had left the faucet running or that a car was honking. On the other hand, it sometimes misclassified sounds (labeling a car driving as running water) or was slow to notify users.
The team is also developing HoloSound, which uses augmented reality to provide real-time captions and other sound information through HoloLens glasses.
