Reut Tsarfaty Profile picture
Jun 29, 2021 12 tweets 5 min read Read on X
Most digital assistants that follow NL instructions focus on uncovering concrete pred-arg structures, such as "book a flight" or "send a mail". However, when humans convey complex instructions they often resort to abstractions, such as emerging shapes, iterations, conditions 1/n
Can #nlproc models learn to interpret such abstract structures, and ground them in a concrete world?

We set out to elicit non-trivial natural lang instructions that contain multiple levels of abstraction, and evaluate models ability to execute them.
arxiv.org/abs/2106.14321

2/n
We do this by means of a collaborative online game, where an instructor receives an image on a 2dim hexagons board, to instruct a remote friend how to draw it on their empty board.

The executer needs to follow the instructions to recreate the image
nlp.biu.ac.il/~royi/hexagon-…

3/n
The key to eliciting abstraction is to design proper stimuli. The images should be: abstract rather than figurative; their description should evoke diverse abstract structures; they should compose different kinds of abstract struct; and reflect increasing levels of complexity 4/n
We designed a gallery of 101 images and elicited above 3K non-trivial NL instructions, reflecting diverse, non-trivial, formal abstract structures expressed in free informal language 5/n
how do #nlproc models do on these? lets look at a typical example. Here a human provided an instruction to recreate the attached image:

“On fifth row from right going leftward, vertically,
color the second tile from the bottom Blue, all tiles
touching it should be Green"
This is the model's result. as you see, it managed to get the direct, predicate-argument like instruction right, correctly positiining the first blue tile.

However, it went downhill from there..
As expected, the model failed to identify the more abstract loop to construct "all tiles touching it".

And this is a case of a very simple image -- you can imagine what happens on the more complex ones. 8/n
Our resulting corpus, we find, is in fact much more interesting than just demonstrating the obvious --- that models fail in complex tasks --- it provides a window into a very rich area of grounded language, that was not tackled and documented before. 9/n
This opens up many research opportunities, on many questions, eg. how do ppl informally express abstraction? What are the diverse ways to *efficiently* convey a complex pattern? Can models "parse" the deeper thought process that reflect the flow underlying the process described?
Data coming soon!

w/ Royi Lachmy @valentina__py

Currently in the corpus:
Current best models:

n/n

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Reut Tsarfaty

Reut Tsarfaty Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @rtsarfaty

Jun 29, 2021
According to Wing 2011, #abstraction in natural language is the act of referring to a set of elements via their shared properties, discarding irrelevant distinctions.

It can be a shape they make ("object"), order of traversal ("control structure") or a condition they all meet>>
From a human communication perspective, speakers make ample use of abstractions expressed in NL, in order to convey complex non-trivial instructions efficiently.

Crucially, the use of such abstract structures in NL is done informally, intuitively, and w/o any formal training >>
Read 10 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(