Thread by @savvyRL

Now that we can write Tiny Papers @iclr_conf, what should we write about?

I'd like to invite all established researchers to contribute Tiny Ideas as inspirations, seeds for discussions & future collaborations! #TinyIdeasForTinyPapers

I'll start. Note: bad ideas == good starts.

1. Calibrate-before-train: before training every model with *data*, train them with noise to calibrate: loss function is to make sure they output "chance probability" — calibrate a model to be as neutral as possible before training starts. Does it help? Why or why not?

2. Does distillation need real data? Can we train student models with *any* data or even noise inputs, just to mimic teacher's behavior? How far does that get us? Is the scaling curve much worse than using real data?

3. Off-manifold-Adv-attack: generative ima models these days seem to be really good at *staying on the image manifold*, eg, whatever you prompt it with, even nonsensical strings, it generates good images. What attacks can get them off the image manifold to generate noises?

Taking a break (off to a meeting) and tagging my favorite researchers to join. @pcastr @bneyshabur @kchonyc @HanieSedghi @mmitchell_ai @jefrankle @sarahookr @hardmaru @laurent_dinh @poolio @dpkingma @doomie @DrJimFan @ChrSzegedy @jekbradbury @andrewgwils @YiTayML @_jasonwei 🥰

4. Positional encoding: I don't have a good idea here, but I just always thought positional encoding as is is quite inelegant😛 the fact that you have to input the index of tokens in addition to an already ordered list seems so unnecessary and ugly. Someone get rid of it please!

5. Single epoch training: there's no real reason a model needs to see any data point twice. Recent large models have turned to single-epoch training but not because we fixed that bug, only because they have too much data. What would it be like to finally fix that bug?