Brain Implant ‘Streams’ a Paralyzed Woman’s Thoughts as If She’s Speaking in Near Real Time

A paralyzed woman can again communicate with the outside world thanks to a wafer-thin disk capturing speech signals in her brain. An AI translates these electrical buzzes into text and, using recordings taken before she lost the ability to speak, synthesizes speech with her own voice.

It’s not the first brain implant to give a paralyzed person their voice back. But previous setups had long lag times. Some required as much as 20 seconds to translate thoughts into speech. The new system, called a streaming speech neuroprosthetic, takes just a second.

“Speech delays longer than a few seconds can disrupt the natural flow of conversation,” the team wrote in a paper published in Nature Neuroscience today. “This makes it difficult for individuals with paralysis to participate in meaningful dialogue, potentially leading to feelings of isolation and frustration.”

On average, the AI can translate about 47 words per minute, with some trials hitting nearly double that pace. The team initially trained the algorithm on 1,024 words, but it eventually learned to decode other words with lower accuracy based on the woman’s brain signals.

The algorithm showed some flexibility too, decoding electrical signals collected from two other types of hardware and using data from other people.

“Our streaming approach brings the same rapid speech decoding capacity of devices like Alexa and Siri to neuroprostheses,” study author Gopala Anumanchipalli at the University of California, Berkeley, said in a press release. “The result is more naturalistic, fluent speech synthesis.”

Bridging the Gap

Losing the ability to communicate is devastating.

Some solutions for people with paralysis already exist. One of these uses head or eye movements to control a digital keyboard where users type out their thoughts. More advanced options can translate text into speech in a selection of voices (though not usually a user’s own).

But these systems experience delays of over 20 seconds, making natural conversation difficult.

Ann, the participant in the new study, uses such a device daily. Barely middle-aged, a stroke severed the neural connections between her brain and the muscles that control her ability to speak. These include muscles in her vocal cords, lips, and tongue and those that generate airflow to differentiate sounds, like the breathy “think” versus a throaty “umm.”

Electrical signals from the outermost part of the brain, called the cortex, direct these muscle movements. By intercepting their communications, devices can potentially decode a person’s intention to speak and even translate signals into comprehensible words and sentences. The signals are hard to decipher, but thanks to AI, scientists have begun making sense of them.

In 2023, the same team developed a brain implant to transform brain signals into text, speech, and an avatar mimicking a person’s facial expressions. The implant sat on top of the brain, causing less damage than surgically inserted implants, and its AI translated neural signals into text at roughly 78 words per minute—about half the rate at which most people tend to speak.

Meanwhile, another team used tiny electrodes implanted directly in the brain to translate 125,000 words into text at a similar speed. A more recent implant with a similarly sized vocabulary allowed a participant to communicate for eight months with nearly perfect accuracy.

These studies “have shown impressive advances in vocabulary size, decoding speeds, and accuracy of text decoding,” wrote the team. But they all suffer a similar problem: Lag time.

Streaming Brain Signals

Ann had a paper-like electrode array implanted on the surface of brain regions responsible for speech. The implant didn’t read her thoughts per se. Rather, it captured signals controlling how vocal cords, the tongue, and other muscles move when verbalizing words. A cable connected the device to a small port fixed on her skull sent brain signals to computers for decoding.

The implant’s AI was a three-part deep learning system, a type of algorithm that roughly mimics how biological brains work. The first part decoded neural signals in real-time. Others controlled text and speech outputs using a language model, so Ann could read and hear the device’s output.

To train the AI, Ann imagined verbalizing 1,024 words in short sentences. Although she couldn’t physically move her muscles, her brain still generated neural signals as if she was speaking—so-called “silent speech.” The AI converted this data into text on a computer screen and speech.

The team “used Ann’s pre-injury voice, so when we decode the output, it sounds more like her,” study author Cheol Jun Cho said in the press release.

After further training that included over 23,000 attempts at silent speech, the AI learned to translate at a pace of roughly 47 words per minute with minimal lag—averaging just a second delay. This is “significantly faster” than older setups, wrote the team.

The speed boost is because the AI processes smaller chunks of neural activity in real time. When given a sentence for the patient to imagine vocalizing—for example, “what did you say to her?”—the system generated both text and vocals with minimal error. Other sentences didn’t fare as well. A prompt of “I just got here” translated to “I’ve said to stash it” in one test.

Long Road Ahead

Prior work mostly evaluated speech prosthetics by their ability to generate short phrases or sentences of just a few seconds. But people naturally start and stop in conversation, requiring an AI to detect an intent to speak over longer periods of time. The AI should “ideally generalize” speech “over several minutes or hours rather than several seconds,” wrote the team.

To accomplish this, they also fed the AI long stretches of brain activity when Ann was not trying to talk, intermixed with those when she was. The AI picked up on the difference—mirroring her intentions of when to speak and when to remain silent.

There’s room for improvement. Roughly half of the decoded words in longer conversations were off the mark. But the setup is a step toward natural communication in everyday life.

Different implants could also benefit from the team’s algorithm.

In another test, they analyzed two separate datasets, one collected from a paralyzed person with electrodes inserted into their brain and another from a healthy volunteer with electrodes placed over their vocal chords. Both could “silent speak” during training and testing. The AI made plenty of mistakes but detected intended speech in near real-time above random chance.

“By demonstrating accurate brain-to-voice synthesis on other silent-speech datasets, we showed that this technique is not limited to one specific type of device,” said study author Kaylo Littlejohn in the release.

Implants with more electrodes to better capture brain activity could improve performance. The team also plans to build emotion into the voice generator to reflect a user’s tone, pitch, and loudness.

In the meantime, Ann is happy with her implant. “Hearing her own voice in near-real time increased her sense of embodiment,” said Anumanchipalli.

The post Brain Implant ‘Streams’ a Paralyzed Woman’s Thoughts as If She’s Speaking in Near Real Time appeared first on SingularityHub.