Thread by @sarah_wandelt

Super excited to share a preprint! At @Caltech, we build the first closed-loop brain-machine (#BMI) interface that is able to decode internal speech 💭. Check it out here: medrxiv.org/content/10.110…

Internal speech decoding is challenging, as it does not have behavioral output like sound or movement. Previously, we found neurons in the supramarginal gyrus (SMG) encoded vocalized speech (pinned tweet). Here, we designed a task to test if it also represents internal speech.

We cued the participant with one of eight written or audio words. He internally said the word in his head and finally vocalized the word. This allowed us to compare internal and vocalized speech in SMG. Excitingly, we found SMG neurons represent internal speech on single trials.

We calculated each neuron’s tuning to individual words. 82-85% of neurons active during internal speech were also active during vocalized speech, and 53-56% were tuned to the same word. This suggests internal and vocalized speech share strong neural representations in SMG.

Decoding of internal speech was also possible! We found robust internal and vocalized speech decoding during an offline analysis. All words were well represented in the neural data, which included lexical words and pseudowords (words without semantic meaning).

Based on those promising results, we built an online internal speech BMI (no vocalized speech was used) that provided feedback by showing the decoded word on the screen. Classification accuracy reached 91% (random chance = 12.5%), even better than during offline analysis!

Important disclaimer, the internal speech BMI is not able to decode people’s private thoughts. It only works if the participant is focusing on a word, and currently only works for words that the decoder had been trained with.

We also decoded several languages in SMG. As our participant is bilingual, we could test how words with different phonetic, but identical semantic content were represented. When training a classifier on both words combined, we found English and Spanish words could be decoded.

In summary, we provided proof-of-concept for an online internal speech BMI from single neuron data in SMG. Those findings are very promising as we only needed ~15 minutes to obtain the training data for our decoder and used a simple linear model.

In the future, we plan to expand the vocabulary size to increase the usability of the device. Massive thanks to co-authors, @davidbjanes, K. Pejsa, B. Lee, C. Liu and R.A. Andersen. If you are at the HSN or @SfNtweets conference next week, come say hi and chat!