very excited to share our paper on reconstructing language from non-invasive brain recordings! we introduce a decoder that takes in fMRI recordings and generates continuous language descriptions of perceived speech, imagined speech, and possibly much more biorxiv.org/content/10.110…
it is generally thought that fMRI responses are too slow for language decoding. to overcome this low temporal resolution, we developed a bayesian decoder that combines state of the art language models and encoding models to generate rapidly changing word sequences (2/7)
we trained and tested our decoder on brain responses while subjects listened to natural narrative stories. given brain responses to new stories that were not used in training, the decoder successfully recovered the meaning of the stories (3/7) 
the same decoder also worked on brain responses while subjects imagined telling stories, even though the decoder was only trained on perceived speech data. we expect that training the decoder on some imagined speech data will further improve performance (4/7)
we pushed the limits of this zero-shot transfer by testing the decoder on brain responses while subjects watched silent movies, and found that the decoder accurately described many movie events. this suggests that our decoder can transfer to non-linguistic semantic tasks! (5/7)
finally, we tested whether our decoder respects user privacy, and found that subject cooperation is currently required to train and to run the decoder. for instance, we developed a set of covert resistance strategies that substantially lowered decoding performance (6/7)
many thanks to the fantastic team @AmandaLeBel3 @shaileeejain @alex_ander that made this possible. we'll be at SNL and SFN over the next few months, so drop by if you'd like to learn more! (7/7)
• • •
Missing some Tweet in this thread? You can try to
force a refresh
