Thread by @RisingSayak

New #Keras example is up on *consistency regularization*or an important recipe for semi-supervised learning and tackling distribution shifts as shown in *Noisy Student Training*.

keras.io/examples/visio…

1/n

https://twitter.com/fchollet/status/1386400929704906754

This example provides a template for performing semi-supervised / weakly supervised learning. A few things one can plug right in:

* Incorporate more data while training the student.
* Filter the high-confidence predictions while training the student.

2/n

The example uses Stochastic Weight Averaging during training the teacher to induce geometric ensembling. With elements like Stochastic Dropout, the performance might even be better.

Here are the full experiments: git.io/JO55v.

I meant *Stochastic Depth*. Apologies for the typo.

By "Filter the high-confidence predictions while training the student" I meant filtering high-confidence predictions from the teacher that would be used while training the student.