A group of researchers from UC Berkeley and UC San Francisco has discovered a method to help individuals with severe paralysis regain naturalistic speech, which is a significant advancement in the field of brain-computer interfaces (BCIs).
The long-standing problem of latency in speech neuroprostheses—the interval between a subject's attempt to speak and the sound that is produced—is resolved by this work. The researchers created a streaming technique that converts brain impulses into audible speech in almost real time using the latest developments in artificial intelligence-based modeling.
“Our streaming approach brings the same rapid speech decoding capacity of devices like Alexa and Siri to neuroprostheses,” said Gopala Anumanchipalli, Robert E. and Beverly A. Brooks Assistant Professor of Electrical Engineering and Computer Sciences at UC Berkeley and co-principal investigator of the study. “Using a similar type of algorithm, we found that we could decode neural data and, for the first time, enable near-synchronous voice streaming. The result is more naturalistic, fluent speech synthesis.”
Related The Future of Cognitive Health
“This new technology has tremendous potential for improving quality of life for people living with severe paralysis affecting speech,” said UCSF neurosurgeon Edward Chang, senior co-principal investigator of the study. Chang leads a clinical trial at UCSF that aims to develop speech neuroprosthesis technology using high-density electrode arrays that record neural activity directly from the brain surface. “It is exciting that the latest AI advances are greatly accelerating BCIs for practical real-world use in the near future,” he said.
The researchers also showed that their approach can work well with a variety of other brain sensing interfaces, including microelectrode arrays (MEAs) in which electrodes penetrate the brain’s surface, or non-invasive recordings (sEMG) that use sensors on the face to measure muscle activity, reports Marni Ellery in Berkeley Engineering.
According to study co-lead author Cheol Jun Cho, who is also a UC Berkeley Ph.D. student in electrical engineering and computer sciences, the neuroprosthesis works by sampling neural data from the motor cortex, the part of the brain that controls speech production, then uses AI to decode brain function into speech.
“We are essentially intercepting signals where the thought is translated into articulation and in the middle of that motor control,” he said. “So what we’re decoding is after a thought has happened, after we’ve decided what to say, after we’ve decided what words to use and how to move our vocal-tract muscles.”
To collect the data needed to train their algorithm, the researchers first had Ann, their subject, look at a prompt on the screen — like the phrase: “Hey, how are you?” — and then silently attempt to speak that sentence.
This latest work brings researchers a step closer to achieving naturalistic speech with BCI devices, while laying the groundwork for future advances.
“This proof-of-concept framework is quite a breakthrough,” said Cho. “We are optimistic that we can now make advances at every level. On the engineering side, for example, we will continue to push the algorithm to see how we can generate speech better and faster.”
