Discover and read the best of Twitter Threads about #BIGbench

Most recents (2)

The release of impressive new deep learning models in the past few weeks, notably #dalle2 from @OpenAI and #PaLM from @GoogleAI, has prompted a heated discussion of @GaryMarcus's claim that DL is "hitting a wall". Here are some thoughts on the controversy du jour. 🧵 1/25
One of @GaryMarcus' central claims is that current DL models fail at compositionality. The assessment of this claim is complicated by the fact that people may differ in how they understand compositionality – and what a "test of compositionality" should even look like. 2/25
Compositionality traditionally refers to a (putative) property of language: the meaning of a complex expression is fully determined by its structure and the meanings of its constituents. (There are good reasons to doubt that language is always compositional in that sense.) 3/25
Read 26 tweets
CALL FOR TASKS CAPTURING LIMITATIONS OF LARGE LANGUAGE MODELS

We are soliciting contributions of tasks to a *collaborative* benchmark designed to measure and extrapolate the capabilities and limitations of large language models. Submit tasks at github.com/google/BIG-Ben…
#BIGbench
All accepted task submitters will be co-authors on the paper releasing the benchmark. Teams at Google and OpenAI will further evaluate BIG-Bench on their best-performing model architectures, across models spanning from tens of thousands through hundreds of billions of parameters.
We encourage submission of tasks by researchers in fields other than computer science which probe the nature of language or intelligence, including: linguistics, cognitive science, philosophy, neuroscience, psychology, animal intelligence, and logic.
Read 5 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!