,
36 tweets,
11 min read
Read on Twitter
Now live-tweeting The Enigma of Neural Text Degeneration as the First Defense Against Neural Fake News by Yejin Choi #neuralgen2019 #naacl2019
Motivating with super funny fake news created which tells she is a co founder of a self driving icecream truck lmao 
Neural Fake news is here 😱😱😱😱
There can be reason for generating fake news 😱
Use talktotransformer.com to generate news based on GPT2 from OpenAI . Can be used for fake news
Mix between conditional and unconditional . Condition on a subset of metadata and generate the body or condition on part of body and guess author and sometimes dont condition on anything . New LM GROVeR
Model is Transformer same as GPT2
Important considerations
How good are these ??
How good does it work on Propaganda 😈 Omg Yejin shows GROVER created language is more prone to be believed as Fake news . What has #neuralgen done !!!!
Size matters as shown . Publically SOTA available discrimator Bert does poorly also smaller models for Grover did better than large on contrast to bert base vs bert large
Lol look at the example
How to generate better text from neural models
See the example generated . Lol
Yejin says that its not cherrypicked . Empirical evidence
Using Beam search does worse lol
Natural distribution of human text differs . Beam search gives artifically high probabilities . Problematic much !!!!
Possible explanation by yejin
She says the more you engage the more repetitions you get . Ugh
Why doesnt maximisation work
Ok hypothesis 3
Why do u think of it??
Interesting insight 😱
Ok now what hppens with top k
More reasaonable
Top-k can still be bland :(
Problem with k sampling portion of tail distribution not of high quality
Possible solution now
Ok works much better
Ok nucleus sampling is better :)
Excellent talk and takeaways from yejins talk
Dont restrict variance
Problems with nucleus sampling :)
Trade off between restrict the variance and dont restrict the variance
Conclusions
