First-of-its-kind technology helps man with ALS ‘speak’ in real time

Table of Contents

Researchers at the University of California, Davis, have developed an investigational brain-computer interface that holds promise for restoring the voices of people who have lost the ability to speak due to neurological conditions.

The participant is enrolled in the BrainGate2 clinical trial at UC Davis Health. His ability to communicate through a computer has been made possible with an investigational brain-computer interface (BCI). It consists of four microelectrode arrays surgically implanted into the region of the brain responsible for producing speech. These devices record the activity of neurons in the brain and send it to computers that interpret the signals to reconstruct voice. Credit: UC Davis Health

In a new study published in the journal Nature, the researchers demonstrate how this new technology can instantaneously translate brain activity into voice as a person tries to speak — effectively creating a digital vocal tract.

Casey Harrell, a 45-year-old man with ALS. Credit: UC David Health

The system allowed the study participant, who has amyotrophic lateral sclerosis (ALS), to “speak” through a computer with his family in real time, change his intonation and “sing” simple melodies.

“Translating neural activity into text, which is how our previous speech brain-computer interface works, is akin to text messaging. It’s a big improvement compared to standard assistive technologies, but it still leads to delayed conversation. By comparison, this new real-time voice synthesis is more like a voice call,” said Sergey Stavisky, senior author of the paper and an assistant professor in the UC Davis Department of Neurological Surgery. Stavisky co-directs the UC Davis Neuroprosthetics Lab.

“With instantaneous voice synthesis, neuroprosthesis users will be able to be more included in a conversation. For example, they can interrupt, and people are less likely to interrupt them accidentally,” Stavisky said.

Decoding brain signals at heart of new technology
#

The man is enrolled in the BrainGate2 clinical trial at UC Davis Health. His ability to communicate through a computer has been made possible with an investigational brain-computer interface (BCI). It consists of four microelectrode arrays surgically implanted into the region of the brain responsible for producing speech.

These devices record the activity of neurons in the brain and send it to computers that interpret the signals to reconstruct voice.

“The main barrier to synthesizing voice in real-time was not knowing exactly when and how the person with speech loss is trying to speak,” said Maitreyee Wairagkar, first author of the study and project scientist in the Neuroprosthetics Lab at UC Davis. “Our algorithms map neural activity to intended sounds at each moment of time. This makes it possible to synthesize nuances in speech and give the participant control over the cadence of his BCI-voice.”

Instantaneous, expressive speech with BCI shows promise
#

The brain-computer interface was able to translate the study participant’s neural signals into audible speech played through a speaker very quickly — one-fortieth of a second. This short delay is similar to the delay a person experiences when they speak and hear the sound of their own voice.

The technology also allowed the participant to say new words (words not already known to the system) and to make interjections. He was able to modulate the intonation of his generated computer voice to ask a question or emphasize specific words in a sentence.

The participant also took steps toward varying pitch by singing simple, short melodies.

His BCI-synthesized voice was often intelligible: Listeners could understand almost 60% of the synthesized words correctly (as opposed to 4% when he was not using the BCI).

Real-time speech helped by algorithms
#

The process of instantaneously translating brain activity into synthesized speech is helped by advanced artificial intelligence algorithms.

The algorithms for the new system were trained with data collected while the participant was asked to try to speak sentences shown to him on a computer screen. This gave the researchers information about what he was trying to say.

The neural activity showed the firing patterns of hundreds of neurons. The researchers aligned those patterns with the speech sounds the participant was trying to produce at that moment in time. This helped the algorithm learn to accurately reconstruct the participant’s voice from just his neural signals.

Clinical trial offers hope
#

“Our voice is part of what makes us who we are. Losing the ability to speak is devastating for people living with neurological conditions,” said David Brandman, co-director of the UC Davis Neuroprosthetics Lab and the neurosurgeon who performed the participant’s implant.

“The results of this research provide hope for people who want to talk but can’t. We showed how a paralyzed man was empowered to speak with a synthesized version of his voice. This kind of technology could be transformative for people living with paralysis.”

Brandman is an assistant professor in the Department of Neurological Surgery and is the site-responsible principal investigator of the BrainGate2 clinical trial.

Limitations
#

The researchers note that although the findings are promising, brain-to-voice neuroprostheses remain in an early phase. A key limitation is that the research was performed with a single participant with ALS. It will be crucial to replicate these results with more participants, including those who have speech loss from other causes, such as stroke.

The BrainGate2 trial is enrolling participants. To learn more about the study, visit braingate.org or contact [mailto:braingate@ucdavis.edu].

Caution: Investigational device, limited by federal law to investigational use.

Funding
#

Program, NIH/National Institute on Deafness and Other Communication Disorders, ALS Association, Simons Collaboration for the Global Brain, NIH/National Institute on Deafness and Other Communication Disorders, Department of Veterans’ Affairs

The paper An instantaneous voice-synthesis neuroprosthesis was published in Nature.

COI Statement
#

Stavisky is an inventor on intellectual property related to speech decoding submitted and owned by Stanford University (U.S. Patent No. 12008987) that has been licensed to Blackrock Neurotech and Neuralink Corp. Wairagkar, Stavisky, and Brandman have patent applications related to speech BCI submitted and owned by the Regents of the University of California (US Patent Application No. 63/461,507 and 63/450,317), including IP licensed by Paradromics Inc. Brandman was a surgical consultant with Paradromics Inc., completing his consultation during the revision period of the manuscript. He is a consultant for Globus Medical Inc. Stavisky is a scientific advisor to Sonera. The MGH Translational Research Center has a clinical research support agreement with Ability Neuro, Axoft, Neuralink, Neurobionics, Paradromics, Precision Neuro, Synchron, and Reach Neuro, for which Hochberg provides consultative input. Mass General Brigham (MGB) is convening the Implantable Brain-Computer Interface Collaborative Community (iBCI-CC); charitable gift agreements to MGB, including those received to date from Paradromics, Synchron, Precision Neuro, Neuralink, and Blackrock Neurotech, support the iBCI-CC, for which Hochberg provides effort. The remaining authors declare no competing interests.