Discover and read the best of Twitter Threads about #TinyIdeasForTinyPapers
Most recents (1)
Now that we can write Tiny Papers @iclr_conf, what should we write about?
I'd like to invite all established researchers to contribute Tiny Ideas as inspirations, seeds for discussions & future collaborations! #TinyIdeasForTinyPapers
I'll start. Note: bad ideas == good starts.
I'd like to invite all established researchers to contribute Tiny Ideas as inspirations, seeds for discussions & future collaborations! #TinyIdeasForTinyPapers
I'll start. Note: bad ideas == good starts.
1. Calibrate-before-train: before training every model with *data*, train them with noise to calibrate: loss function is to make sure they output "chance probability" — calibrate a model to be as neutral as possible before training starts. Does it help? Why or why not?
2. Does distillation need real data? Can we train student models with *any* data or even noise inputs, just to mimic teacher's behavior? How far does that get us? Is the scaling curve much worse than using real data?
