Thread by @LChoshen

Are all language orders as hard?
Supposedly, for RNNs yes, for Transformers no

@JenniferCWhite @sleepinyourhat
aclanthology.org/2021.acl-long.…

github.com/rycolab/artifi… (currently empty)
#NLProc
Really cool, with a caveat

The paper creates synthetic languages (using a PCFG) with various ordering rules, being able to compare each order.

They also add agreement, and a vocabulary to introduce more of real language important features (e.g. long-distance dependencies)

Last, the words are taken from words that could have been in English (e.g. daxing)

Their results are the Transformers do have inductive biases, but those are not towards the most probable language orders (e.g. VOS is quite easy for Transformers)

Another interesting thing is that RNNs do not show that. Despite all that we know about their strong inductive bias regarding forgetting (hard to relate far away words\information).

Last, it seems that a lot of what makes things easy and hard is related to the agreement, at least from looking at which rules matter together.

I really like this direction (which they seem to discuss a lot, seems like a reviewer was quite annoying) and the results are interesting.

However, honestly, I also have a lot of criticism about this work.

A lot of the details are missing. For example, how were the words tokenized? If BPE, then morphology is split to tokens perfectly? If none, then their sophisticated way of choosing words doesn't matter, because they anyway convert the words into one hot vectors?

How large were the vocabularies? What were the parameters of the networks train? There are works saying that depth changes inductive biases and not only arch...

Because of the large number of networks (~128 * 10 reps), they train on really small amounts of data. Hard to say to which amount did that matter, and whether with a small vocab it is reasonable. Still, hard to generalize from the results because of it.

Last, and that is especially peculiar:
"Weights given to each production were chosen manually through experimentation. Some principles for choosing weights for a grammar in this manner are described by Eisner and Smith (2008)"

So, we can't know what type of sentences they create (balanced trees, long sentences, really low weights on something that makes a phenomenon\switch less interesting). And they don't share the method or even the chosen weights.

To sum, this is a very interesting approach, with a lot of potential, that could benefit from more analysis of the results (e.g. per switch), and analysing myself is also problematic as a lot is left out of the paper (nor there is an appendix with the info.)

@threadreaderapp unroll