Overview
- The study, published Monday in Nature Neuroscience, shows a single high-density electrocorticography implant can run parallel speech and gesture decoders and render text plus a personalized full-body avatar in real time.
- Researchers implanted a 253-electrode subdural ECoG array over sensorimotor cortex and trained participant-specific neural networks to translate attempted phrases and communicative gestures into commands for on-screen text and avatar motion.
- The trial enrolled three people with severe paralysis and tested simultaneous speech-and-gesture decoding in two participants, with real-time accuracies that varied by person and task—for example, one participant showed about 66% gesture and 70% speech accuracy in live concurrent tests while a second participant posted median 100% in limited conversational trials.
- Models trained on both isolated and concurrent attempts performed best; hybrid cross-modality training improved generalization across contexts and sharply reduced false activations of the opposite modality.
- The work is a proof of concept with clear limits: it used wired external processing, a small cohort, roughly ten phrases and gestures per participant, and short structured tasks, and the team says NIH-funded follow-ups will build wireless implants, expand vocabularies, and test larger cohorts before clinical use.