A list of predictions for 2023 for the field of LLMs🧵
There will be an opensource Chinchilla-style LLM released this year at the level of text-davinci-*. Maybe not from the ones we expect🤔This will obliterate ChatGPT usage and enable various types of fine-tuning / soft-prompting and cost/speed improvements.
(as per above) ChatGPT will remain free until it dies out. Replaced by a paid-for equivalent based on (a) better model(s). The balance of the force will be restored at this stage. So it's not a so silly strategy to build a product for free on ChatGPT 🤔... dangerous game still.
AI assistants will pop-up in every existing product silo. Existing AI-first company defensibility will be battle-tested, big time.
AGI does not pop up yet (even if one of the big labs have an internal LLM AlphaZero moment in 2023). Investor craziness dies out, short-context LLM assistants (as per above #1 #3) are heavily commoditized. Still 1-2 massive companies start to emerge in the B2B space #product
As users, we stop "experiencing" the magic of scale as larger and larger models are released (after all it's a power law).
Context-size becomes the clear limiting factor to many use-cases. OpenAI releases a massive context-size model in 2023 and create a moat for 6 months, time for the opensource model to catch-up and even prevail, likely with a memorizing transformer style approach.
"Memorizing transformer style" or similar approach become predominant as they are always up-to-date and infinite in size. They enable a new wave of products that can operate on entire companies data or entire codebases, or entire chat history.
By the end of 2023 at least one state-actor start a program to compete with OpenAI, Meta AI and Google/DeepMind.
Oh and we finally get good papers on "science of fine-tuning", aka, scaling laws on fine-tuning based on pre-training / fine-tuning compute and pre-training / fine-tuning data size
• • •
Missing some Tweet in this thread? You can try to
force a refresh
