How can we choose examples for a model that induce the intended behavior?

We show how *active learning* can help pretrained models choose good examples—clarifying a user's intended behavior, breaking spurious correlations, and improving robustness!

arxiv.org/abs/2204.08491

1/ A fundamental challenge in ML is *task ambiguity*: when the training data doesn't specify the user's intended behavior for all possible inputs

For example, imagine you have a dataset of red squares and blue circles. How should the model classify blue squares?

2/

One of the reasons I think GPT-J is so cool is that its pretraining data is publicly available

This lets us ask questions that were impossible to answer for LLMs like GPT-3

For example: "did our model actually learn the task or was this example in the training data?"

1/ Case in point, a recent paper looks at few-shot performance on numerical tasks like arithmetic

arxiv.org/abs/2202.07206
by @yasaman_razeghi @rloganiv @nlpmattg @sameer_

2/

DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning

SSL is a promising technology, but current methods are field-specific. Can we find general algorithms that can be applied to any domain?

🌐: dabs.stanford.edu
📄: arxiv.org/abs/2111.12062

🧵👇 #NeurIPS2021

1/ Self-supervised learning (SSL) algorithms can drastically reduce the need for labeling by pretraining on unlabeled data

But designing SSL methods is hard and can require lots of domain-specific intuition and trial and error

2/

Love the "data science maturity levels" in @Patterns_CP

Interesting way to contextualize research at a glance (reminds me a bit of @justsaysinmice)

Full list in thread: 1) Concept

Basic principles of a new data science output observed and reported (e.g., statement of principles, dataset, new algorithm, new theoretical concept, theoretical system infrastructure)

A quick thread for PhD admits thinking about potential advisors:

I see a lot of discussion about "hands-on" vs "hands-off" advisors

But I think there are at least 3 underlying dimensions here, each of which is worth considering in its own right:

👇 [THREAD]

1/ 1) Directiveness—how much your advisor directs your research, in terms of the problems you work on or day-to-day activities

2/

Some takeaways from @openai's impressive recent progress, including GPT-3, CLIP, and DALL·E:

[THREAD]

👇1/ 1) The raw power of dataset design.

These models aren't radically new in their architecture or training algorithm

Instead, their impressive quality is largely due to careful training at scale of existing models on large, diverse datasets that OpenAI designed and collected.

2/