Thread by @EnricoShippole

Releasing Yarn-Llama-2-13b-128k, a Llama-2 model, trained for 128k context length using YaRN scaling. The model was trained in collaboration with u/bloc97 and @theemozilla of @NousResearch and @Void13950782 of @AiEleuther.

The model can be found on @huggingface here: huggingface.co/conceptofmind/…

We worked to extend the context length of the Llama-2 13b and 7b models through fine-tuning. The model passes all our evaluations and maintains the same perplexity at 128k extrapolation surpassing the performance of our other recent methodology, NTK-part scaling.

A Yarn-Llama-2-7b model trained for 128k context length is available on @huggingface here: huggingface.co/conceptofmind/…

The models have similar performance to the base LLaMA 2 models on the Open LLM benchmarks while scaling context length directly to 128k.

We also trained a set of models at 64k context length. You can find the Yarn-Llama-2-13b-64k model here: huggingface.co/conceptofmind/…

As well as the Yarn-Llama-2-7b-64k model here: huggingface.co/conceptofmind/…

We are releasing all of the code, open-source, to fully reproduce the results of the paper. The repository containing u/bloc97 and @theemozilla’s implementation of YaRN rotary embeddings can be found here: github.com/jquesnelle/yarn

I would also recommend checking out the phenomenal research by @OfirPress on ALiBi which laid the foundation for many of these scaling techniques: arxiv.org/abs/2108.12409

It is also worth reviewing the paper, A Length-Extrapolatable Transformer, and xPos technique which also applies scaling to rotary embeddings: arxiv.org/pdf/2212.10554…

We previously trained the first publicly available model with rotary embedding scaling here:

https://twitter.com/EnricoShippole/status/1655599301454594049?s=20

You can find out more about the @NousResearch organization here: huggingface.co/NousResearch

The compute for these model releases is all thanks to the generous sponsorship by @carperai, @EMostaque, and @StabilityAI. This is not an official @StabilityAI product. Thank you to @dmayhem93 and @jonbtow as well for helping.

A big thank you to @Void13950782 and @AiEleuther for facilitating the discussions about context-length extrapolation and helping to write the paper. Truly an awesome open-source team and community.

If you have any questions about the methodology and models be sure to reach out to @theemozilla and me! We will try to respond promptly.

All of the models can be found on Huggingface: huggingface.co/conceptofmind

The models used @tri_dao's flash attention 2 and part of @togethercompute's codebase. You can find out more about Flash Attention 2 here: github.com/Dao-AILab/flas…

Thank you to @AlpinDale and @pygmalion_ai for providing resources to help run evaluations on these models.