Thread by @deliprao

Despite the amazing results I’ve experienced with ChatGPT, this is not a correct way to look at LLM vs. Google search. Since several other tweets have made this equivalence and have been eager to spell doom for Google, let’s examine the details:

https://twitter.com/jheitzeb/status/1598154943512596480

1. Google has more LLMs deployed internally than any place I know. If private communication is to be believed that number is in the order of “few dozens”. Not talking of BERT/T5 sized models here.

2. Google also has more compute than anyone. The joke is only NSA probably has an estimate of Google’s compute. So they are not compute-limited to build as much big a model as they want.

3. Google neither has lack of talent to build that. Together with Deepmind, Brain probably corners most of the AI talent today. Yes it’s true that many folks on the attention paper left to build startups, but that’s just opportunistic and not a sign of affairs at Google.

4. It is very very likely the largest/most performant model in industry today is either at Google or Deepmind. (Baidu is Chinese government)

5. So natural question is, why then Google has not replaced the traditional search page with a GPTChat like interface, sticking an ad underneath? For many reasons:

6. Search satisfaction is a reputation business. Once folks are let down by a search result on something critical, they will use less and less of that search interface. For example, Twitter search on twitter.com sucks, so I use other options to lookup my own tweets.

7. LLMs despite their recent impressive performances, still hallucinate a lot, and are temperamental to starting conditions. LLMs are less trust worthy in that sense.

8. Most importantly, LLMs fail in strange unpredictable ways. We are okay with things that fail and learn how to adapt to them as long as they fail predictably.

9. For instance, in early days of Google, they did not answer natural language queries. So humans adapted with query rewriting strategies to satisfy their intent. Google, for their credit, recognized this and doubled down on implementing natural language query answering.

10. Search users have multiple intents. Besides information seeking, browsing intent has not gone anywhere. The internet, specifically Dmoz/Yahoo directory, killed Yellow Pages. Google killed internet directories by generating an intent specific directory, aka search results.

11. We haven’t abandoned browsing behavior, instead we have readjusted search expectations to be somewhere between information seeking and browsing. This is also why Google shows both a “knowledge panel” and a list of search results.

12. Google’s knowledge panel today is powered with a combination of language models, ontologies, and an array of information extraction techniques. But all of them are able to not just answer your question but also list the sources they come from.

13. For many queries, this provenance information is as valuable, if not more, to the user as the exact answer. LLMs cannot, in the traditional formulation of an LM, reliably tell where the answers are coming from.

14. In addition, Google generates related questions to satisfy query refinements rapidly.

15. In fact, when they have the information, google is so eager to satisfy your intent that they don’t even want you to enter their result page with ads. This is because they understand users well. People will abandon if things are not convenient.

16. All this said, LLMs and even small LMs like BERT can dramatically change Google’s search traffic. These open LMs (thanks to @huggingface), make it easy for a lot of site-specific search interfaces to improve. But that’s a small fraction of google searches.

17. Interestingly, these LLM-based answering engines will kill traffic to a lot of small and and not-so-small websites. If this product becomes widely accessible and free, traffic to sites like Wikipedia will drop. So will a lot of how-to websites and other niche websites.

18. By now it should be abundantly clear that Google is not going anywhere. The rumors of its death are clearly exaggerated. It is not for the lack of resources or talent that Google has not turned on a ChatGPT-like interface.

19. Google understands search better than anyone. But funnily they are not a search company. They are a company seeking to understand and organize the world’s knowledge. So far they have plenty of products demonstrating that, with search engine being just one of them.

20. This opinion of ChatGPT “killing” search is not something most experienced NLP and IR researchers share. It’s typically held by a startup folks pushing products or investors pushing their portfolio cos or >>

by developers who think we don’t need other #nlproc methods because we can prompt large language models. While this a very critical or “foundational” technology, a better way to look at these things is they are complementary tools. /fin