Researchers at the University of California, San Francisco have demonstrated a brain implant that decodes intended speech and gestures together, enabling people with paralysis to communicate through a digital avatar. Nature Portfolio reported the advance on September 15, following publication of the study in Nature Neuroscience on September 14.
The team led by Edward Chang recorded brain activity in three participants with impaired speech and movement. According to the paper, two participants took part in the avatar experiments: one had a brainstem stroke and the other had amyotrophic lateral sclerosis. The system displayed decoded speech as text while a personalized full-body avatar performed gestures.
A surgically placed array of 253 electrodes on the brain surface captured signals associated with speech and upper-body movements. Machine-learning models translated those signals into computer commands. Participants attempted phrases and gestures separately and together; the simultaneous experiments used five phrases and four gestures for one participant, and ten of each for the other.
The researchers found that training on speech or gestures alone did not transfer reliably to doing both together, because the neural activity patterns partly overlap and change with the task. Training with both isolated and simultaneous attempts improved performance. Additional training helped prevent either decoder from activating when only the other form of communication was intended.
The findings establish a proof of concept for more expressive communication, the authors said. They do not establish unrestricted conversation: the study used small sets of phrases and gestures, and the conversational demonstrations involved limited trials. The paper calls for larger participant groups and broader vocabularies to assess how well the approach can scale.
The US National Institutes of Health, which funded the research, said the study used wires linking implanted sensors to external processors. Chang said his team would next test a fully implantable wireless version. That is a planned step, while the published results concern the wired experimental system and its potential to combine words with body language.
Comments