Thread by @theshawwn

Suppose you wanted to train a world-class GPT model, just like OpenAI. How? You have no data.

Now you do. Now everyone does.

Presenting "books3", aka "all of bibliotik"

- 196,640 books
- in plain .txt
- reliable, direct download, for years: the-eye.eu/public/AI/pile…

thread 👇

I wrote up some details here: github.com/soskek/bookcor…

In OpenAI's papers on GPT-2 and 3, you'll notice references to datasets named "books1" and "books2".

books1 appears to be bookcorpus, or similar.

But OpenAI will not release information about books2; a crucial mystery.

We suspect OpenAI's books2 dataset might be "all of libgen", but no one knows. It's all pure conjecture.

Nonetheless, books3, released above, is "all of bibliotik", which I imagine will be of interest to anyone doing NLP work. Or anyone who wants to read 196,640 books. :)

You now have OpenAI-grade training data at your fingertips. books3 is a direct download to 200k plaintext books, preprocessed via ftfy.fix_text(), as OpenAI did.

More importantly, we anticipate that the download link will remain stable, possibly for years to come.

This stable link is thanks to a community of data enthusiasts: the-eye.eu. They gather and make available data "for the benefit of humankind".

They have an interesting DMCA policy. I urge you to read it: the-eye.eu/dmca.mp4

(Did you read their DMCA policy yet? the-eye.eu/dmca.mp4 I almost didn't bother, so I better call it out twice to make sure you don't miss the crucial details. Please, take a moment to review it; you'll find it well-worth your time.)

I can't stress enough how unbelievably impactful this feels. Imagine finally not having to worry about "how do I get imagenet?" or "Where did Celeb-A go?" or "I want to make GPT-3; where is the data?"

In hindsight, a community of data hoarders was a natural fit.

Now, 200k books is all well and good, but what if you're not happy with it? What if you want more?

What if you want to make a GPT model that knows literally everything about programming?

answer: train on "literally all of github".

Download: the-eye.eu/public/AI/pile…

A 100GB github.tar isn't quite "all of github." That's not scientifically accurate to claim. But "all of github" wouldn't necessarily be much more useful than what you get there. 100GB compressed is a pretty good chunk.

Something like 12GB of C++, 7GB of .py, 4GB of .js.

There's more to say, but this is enough for now. I never know ahead of time whether any given tweet will do well, so this is always a lottery. It *feels* like I should be shouting to everyone about this – who wouldn't want 200,000 books? – but who knows.

Spread it like 🔥🔥

Patreon: patreon.com/shawwn

ML discord (~850 members): discordapp.com/invite/x52Xz3y

I made a script to let you mount books3 remotely. It only takes a few minutes to run, and uses less than 1GB:

https://twitter.com/theshawwn/status/1320871642634080262?s=21

Now you don’t need a big hard drive or lots of patience just to poke around the dataset. I think you can start looking at books in < 10 min.

https://twitter.com/theshawwn/status/1320871642634080262