Discover and read the best of Twitter Threads about #BIGbench
Most recents (2)
The release of impressive new deep learning models in the past few weeks, notably #dalle2 from @OpenAI and #PaLM from @GoogleAI, has prompted a heated discussion of @GaryMarcus's claim that DL is "hitting a wall". Here are some thoughts on the controversy du jour. 🧵 1/25
One of @GaryMarcus' central claims is that current DL models fail at compositionality. The assessment of this claim is complicated by the fact that people may differ in how they understand compositionality – and what a "test of compositionality" should even look like. 2/25
Compositionality traditionally refers to a (putative) property of language: the meaning of a complex expression is fully determined by its structure and the meanings of its constituents. (There are good reasons to doubt that language is always compositional in that sense.) 3/25
CALL FOR TASKS CAPTURING LIMITATIONS OF LARGE LANGUAGE MODELS
We are soliciting contributions of tasks to a *collaborative* benchmark designed to measure and extrapolate the capabilities and limitations of large language models. Submit tasks at github.com/google/BIG-Ben…
#BIGbench
We are soliciting contributions of tasks to a *collaborative* benchmark designed to measure and extrapolate the capabilities and limitations of large language models. Submit tasks at github.com/google/BIG-Ben…
#BIGbench
All accepted task submitters will be co-authors on the paper releasing the benchmark. Teams at Google and OpenAI will further evaluate BIG-Bench on their best-performing model architectures, across models spanning from tens of thousands through hundreds of billions of parameters.
We encourage submission of tasks by researchers in fields other than computer science which probe the nature of language or intelligence, including: linguistics, cognitive science, philosophy, neuroscience, psychology, animal intelligence, and logic.
