Brain Implant Lets a Paralyzed Patient Speak and Gesture at the Same Time

A brain implant can now do something that wasn’t possible before. It can translate a person’s attempted words and gestures at the same time.

Researchers at the University of California, San Francisco, tested the system in two people with severe paralysis. As the participants tried to speak, wave, nod or combine the two, the implant recorded activity from the brain’s movement centers and converted it into text and movements on a virtual avatar.

The challenge was that the brain did not produce exactly the same patterns when speech and gestures happened together as it did when each was attempted alone. A decoder trained only on isolated words or movements struggled with the combinations. It worked better once researchers trained it on both situations.

We all know that a conversation between two people is more than just the spoken word. The Italians, as they say, talk with their hands. As such, the whole idea behind this brain-computer interface is not just to restore a person’s words but also some semblance of their body language.

“Conversation is about much more than the words being spoken. It’s a multilayered, dynamic process involving the whole motor cortex,” Edward Chang, a neurosurgeon at the University of California, San Francisco, and senior author of the study, said in a statement. “This proof-of-concept shows us it’s possible for a BCI to restore some of this freedom and flexibility.”

More Than the Sum of Its Parts

The researchers enrolled three people with severe speech and/or movement impairments in UCSF’s BRAVO clinical trial. Two had experienced brainstem strokes; the third had amyotrophic lateral sclerosis, or ALS (the same disease Stephen Hawking had). Each person had a grid of 253 electrodes implanted on the surface of the left side of the brain, covering regions involved in speech and movement.

The main speech-and-gesture experiment involved two participants because the third withdrew before the full-body avatar experiments began. One participant, who had paralysis after a stroke, silently attempted five phrases and four gestures. The participant with ALS attempted 10 phrases while vocalizing and imagined 10 gestures, including waving, shrugging, and giving a thumbs-up. The researchers tested all 100 possible phrase-and-gesture combinations for this participant.

Some electrodes responded mainly to speech and others mainly to gestures. But many responded to both, particularly in the precentral gyrus—a strip of cortex involved in controlling movement. Speech activity tended to cluster farther down this region and gesture activity farther up, with a substantial area of overlap between them.

It’s precisely this overlap that has caused previous similar devices to fail. When researchers trained a machine-learning decoder only on speech performed alone or gestures performed alone, its performance suffered when participants tried both simultaneously.

Training it on examples of both isolated and simultaneous behavior worked better. The neural patterns evidently shifted enough during multitasking that the decoder needed to experience those contexts rather than assume that “speech plus gesture” was simply two independent signals laid on top of each other.

Encouragingly, the system could also decode speech-and-gesture combinations it had never encountered together during training. That’s a big plus for any future device as nobody could realistically train a BCI on every possible pairing of a word with a gesture.

From Words to a Whole Body

The researchers then connected the decoders to personalized full-body avatars. Intended speech appeared as text while decoded gestures made the avatar wave, clap, shrug, or perform other movements in real time.

During simultaneous testing in the participant with ALS, the system correctly classified speech 70% of the time and gestures 66% of the time. In a small conversational demonstration, those figures rose to 75 and 85%, respectively. The stroke participant reached 100% for both channels in conversational testing, although that result came from only three tiny blocks of trials.

Previously, in 2023, Chang’s group demonstrated a BCI that converted attempted speech into text, synthetic speech, and facial movements on an avatar. Other systems have restored cursor control and, more recently, generated synthetic speech almost instantaneously from brain activity. The new work pushes in a different direction: getting several forms of communication to coexist.

“But most of these studies have focused on restoring one of these functions at a time,” Samantha Brosler, a UCSF bioengineering researcher and study co-author, told Nature.

There are major caveats. The experiment involved only two people for simultaneous decoding, used tiny vocabularies and discrete gestures rather than free-flowing conversation, and relied on wires running from the implanted array to external equipment. The authors say larger vocabularies, continuous movements, and faster decoding remain future goals.

Still, the experiment suggests that restoring communication may eventually mean more than giving someone their words back. It could mean restoring body language.

The study was published in the journal Nature Neuroscience.