JUST IN: Google DeepMind releases Gemma, a series of open models inspired by the same research and tech used for Gemini.
Open models fit various use cases so this is a very smart move from Google.
Great to see that Google recognizes the importance of openness in AI science and technology.
There are 2B (trained on 2T tokens) and 7B (trained on 6T tokens) models including base and instruction tuned versions. Trained on a context length of 8192 tokens.
Commercial use is allowed.
These are not multimodal models but based on the reported experimental results they appear to outperform Llama 2 7B and Mistral 7B.
I am excited about those MATH, HumanEval, GSM8K, and AGIEval results. These are really incredible results for a model this size.
Excited to dive deeper into these models. The model prompting guide is dropping soon. Stay tuned!
When I said outperforms other models, I meant generally outperforms them on all the benchmarks. Llama has a lot of catching up to do but it is interesting to see Mistral 7B trail Gemma very closely. These numbers don't really mean much in the context of real-world applications.
If you follow me here on X, you know how excited I get about unlocking unique value and use cases with small language models (SLMs). It will also be fun to run these locally and other other small devices. As I have been saying, SLMs are underexplored. It's a mistake to just see them as research artifacts.
Google DeepMind just announced Gemini, their largest and most capable AI model.
A short summary of all you need to know:
1) What it is - Built with multimodal support from the ground up. Remarkable multimodal reasoning capabilities across text, images, video, audio, and code. Nano, Pro, and Ultra models are available to support different scenarios such as efficiency/scale and support complex capabilities.
2) Performance - The results on the standard benchmarks (MMLU, HumanEval, Big-Bench-Hard, etc.) show improvement compared to GPT-4 (though not by a lot). Still very impressive!
3) Outperforming human experts - They claim that Gemini is the first model to outperform human experts on MMLU (Massive Multitask Language Understanding), a popular benchmark to test the knowledge and problem-solving abilities of AI models.
4) Capabilities- Gemini surpasses SOTA performance on a bunch of multimodal tasks like infographic understanding and mathematical reasoning in visual contexts. There was a lot of focus on multimodal reasoning capabilities with the ability to analyze documents and uncover knowledge that's hard to discern. The model capabilities reported are multimodality, multilinguality, factuality, summarization, math/science, long-context, reasoning, and more. It's probably one of the most capable models by the looks of it.
5) Trying it out - Apparently, a fine-tuned Gemini Pro is available to use via Bard. Can't wait to experiment with this soon.
6) Availability - Models will be made available for devs on Google AI Studio and Google Cloud Vertex AI by Dec 13th.
blog:
technical report:
Here is the model verifying a student's solution to a physics problem. Huge implications in education. Will be taking a very close look at applications here.
As is becoming common now, very little to no details on architecture but it's great to see distillation useful for the Nano series models.
It provides an AI chat-based assistant within the Jupyter environment that allows you to generate code, summarize content, create comments, fix errors, etc.
You can even generate entire notebooks using text prompts!
MPT-30B training data. "To build 8k support into MPT-30B efficiently, we first pre-trained on 1T tokens using sequences that were 2k tokens long, and continued training for an additional 50B tokens using sequences that were 8k tokens long."