,
5 tweets,
2 min read
Read on Twitter
My postdoc, PhD student and RA have a cool new paper on arXiv, at CICLing this Tuesday at 12h10, & as poster on Tuesday at the MPI Nijmegen
Lisa Beinborn, Samira Abnar, Rochelle Choenni (2019): Robust evaluation of language-brain encoding experiments, arxiv.org/abs/1904.02547
1/5
Lisa Beinborn, Samira Abnar, Rochelle Choenni (2019): Robust evaluation of language-brain encoding experiments, arxiv.org/abs/1904.02547
1/5
It's part of a new research line, where we study how well neural language models allow us to predict brain activity -- *cool* because we learn more about the cognitive plausibility of our models!
Eg, in early 2018, we studied how well 8 word embeddings models predict fMRI.
Eg, in early 2018, we studied how well 8 word embeddings models predict fMRI.
But it is also *hard*, because we need few best practices have been established yet.
Previous encoding models have typically only been evaluated on a single datasets (& are likely overtuned), code is not published, evaluation measures have not been standardised.
Previous encoding models have typically only been evaluated on a single datasets (& are likely overtuned), code is not published, evaluation measures have not been standardised.
Lisa, Samira & Rochelle now use 4 datasets, a constant encoding model, constant experimental parameters, a reasonable baseline AND compare results for multiple metrics & parameters. Results, in many cases, are not
significantly different from chance. Disappointing, but important.
significantly different from chance. Disappointing, but important.
Their code is on github: github.com/beinborn/brain…
The 2018 paper is also on arXiv: arxiv.org/abs/1711.09285
And an associated demo online:
projects.illc.uva.nl/LaCo/clclab/ne…
The 2018 paper is also on arXiv: arxiv.org/abs/1711.09285
And an associated demo online:
projects.illc.uva.nl/LaCo/clclab/ne…
