Thread by @martin_gorner

Google's LaMDA paper arxiv.org/abs/2201.08239 shows yet another information retrieval strategy: it has been taught to ask a search engine, or a calculator .

The first answer "It [Eiffel Tower] was constructed in 1887" is generated directly, but also recognized as containing a factual statement. This sends the whole context to LaMDA-Research which is trained to generate search queries, here "TS, Eiffel Tower, construction date"

"TS" means "ToolSet", i.e. the generated text is meant for a tool, the search engine, not the user.
Info from the search, i.e. "Eiffel Tower / construction started: 28 jan 1887" is appended to the context, which is sent to LaMDA-Research again.

And LaMDA-Research responds with "TS, Eiffel Tower, completed when" which is again a search engine query (prefix "TS"). It is trained to issue follow up queries if necessary.

All the retrieved info is added to the context and the context re-evaluated through LaMDA-Research, until it is finally happy and issues a response prefixed with "user" instead of "TS": "User, work started on it in Jan 1887 and it was opened in March 1889."

Contrast this with:
1) traditional NLP models (facts stored in trained weights directly)
2) specialised "memory layers" for facts like arxiv.org/pdf/1907.05242…. Specific learnable weights are used as an information database with a learned retrieval mechanism.

3) DeepMind's RETRO (deepmind.com/blog/article/l…) where the model is trained to retrieve relevant documents from a pile of text by embedding proximity, and add them to the context to generate better answers.

So much interesting innovation !
Some way still to go. The LaMDA authors have thankfully included some entertaining failure modes in their article 😁.