Open-Science NLP Bounty: ($100 + $100 to charity)
Task: A notebook demonstrating experiments within 30(!) PPL (<84) of this widely cited LM baseline on PTB / WikiText-2 using any non-pretrained, word-only Transformer variant.
Context:
Task: A notebook demonstrating experiments within 30(!) PPL (<84) of this widely cited LM baseline on PTB / WikiText-2 using any non-pretrained, word-only Transformer variant.
Context:
The state of benchmarking in NLP right now is so strange. These goofy websites keep precisely-curated leaderboards (paperswithcode.com/sota/language-…), and hardworking grad students cannot get within 2x! these reported results.
(Also not calling out this paper or authors in particular. Just a general frustration on replicability.)
Winning response. Props for author for responding.
I will accept other submissions if others are motivated to find a different solution.
Very interesting explanation for why this is so difficult, and why it should arguably not be used in the future.
I will accept other submissions if others are motivated to find a different solution.
Very interesting explanation for why this is so difficult, and why it should arguably not be used in the future.
@JesseDodge @ssgrn @nlpnoah Just out of curiosity What's the over-under on how many GPU hours this would have taken to replicate manually?
arxiv.org/abs/1909.03004
arxiv.org/abs/1909.03004
