Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Oskar Hallström Profile picture
Oskar Hallström
@oskar_hallstrom
AI R&D Engineer @lightonio. Former Indie One Hit Wonder @ Billie Garlic
Dec 19, 2024 • 10 tweets • 3 min read
torch.randperm isn't fully random and it can bias your trillion token-training run 😱

Today we're releasing ModernBERT, a new SOTA encoder-only model series. In this thread however, I'll share how torch.randperm (temporarily) put a wrench in the works (1/10) Image Background: In native PyTorch data samplers, torch.randperm is used to select the order of which the samples of a dataset should be retrieved when shuffling is used. torch.randperm(n) generates a random permutation of ints from 0 to n-1. (2/10)