Thread by @Replit

At #ReplitDevDay, we announced we’ve trained and are open-sourcing our first Complete Code model.

Introducing replit-code-v1-3b:

- 2.7B params
- 20 languages
- 525B tokens
- 40% better than comparable models
- Trained in 10 days

Take a look at the benchmarks yourself 🧵

Replit-code-v1-3b & replit-finetuned-v1-3b were trained entirely on code and were meant for single-line code completion.

We didn’t expect either to perform so well on HumanEval, but they did.

replit-finetuned-v1-3b outperformed all OSS code models, even those 5x its size.

Both models also benchmark impressively well against commercial models.

replit-finetuned-v1-3b is by far the smallest model on the table and it outperformed Codex and LLaMA.

PaLM-Coder is 200x larger and we’re closing in on their performance with much better latency.

We also noticed a surprising capability in non-coding reasoning, despite the model being trained entirely on code.

We benchmarked against models trained for reasoning tasks and replit-code-v1-3b performed incredibly well.

The important piece of info to remember here is that we trained this model in under 10 days.

The training was even called internally “YOLO RUN”.

We’re excited about these benchmarks but can do even better.

We've already started work to train a 7B model.