Thread by @mishig25

How do language models (like BERT or GPT) "see" words?

TLDR: whereas we see 𝚆𝚎̄𝚕𝚌𝚘́𝚖𝚎̂ 𝚝𝚘́ 𝚝𝚑𝚎̈ 🤗 𝚃𝚘̂𝚔𝚎́𝚗𝚒̄𝚣𝚎̄𝚛𝚜, language models see [𝟷0𝟷, 𝟼𝟷𝟼0, 𝟸000, 𝟷𝟿𝟿𝟼, 𝟷00, 𝟷𝟿𝟸0𝟺, 𝟷𝟽𝟼𝟸𝟿, 𝟸0𝟷𝟻, 𝟷0𝟸]
🧵 on Tokenization by examples
1/

2/ NLP Tokenization steps are ↳ 𝚗𝚘𝚛𝚖𝚊𝚕𝚒𝚣𝚊𝚝𝚒𝚘𝚗 ➜ 𝚙𝚛𝚎-𝚝𝚘𝚔𝚎𝚗𝚒𝚣𝚊𝚝𝚒𝚘𝚗 ➜ 𝚖𝚘𝚍𝚎𝚕 ➜ 𝚙𝚘𝚜𝚝-𝚙𝚛𝚘𝚌𝚎𝚜𝚜𝚒𝚗𝚐.

Together, they are called a "tokenization pipeline"
huggingface.co/docs/tokenizer…

3/ 𝚗𝚘𝚛𝚖𝚊𝚕𝚒𝚣𝚊𝚝𝚒𝚘𝚗:
𝚆𝚎̄𝚕𝚌𝚘́𝚖𝚎̂ 𝚝𝚘́ 𝚝𝚑𝚎̈ 🤗 𝚃𝚘̂𝚔𝚎́𝚗𝚒̄𝚣𝚎̄𝚛𝚜 ➜ 𝚆𝚎𝚕𝚌𝚘𝚖𝚎 𝚝𝚘 𝚝𝚑𝚎 🤗 𝚃𝚘𝚔𝚎𝚗𝚒𝚣𝚎𝚛𝚜

4/ 𝚙𝚛𝚎-𝚝𝚘𝚔𝚎𝚗𝚒𝚣𝚊𝚝𝚒𝚘𝚗:
𝚆𝚎𝚕𝚌𝚘𝚖𝚎 𝚝𝚘 𝚝𝚑𝚎 🤗 𝚃𝚘𝚔𝚎𝚗𝚒𝚣𝚎𝚛𝚜 ➜ [('𝚆𝚎𝚕𝚌𝚘𝚖𝚎', (0, 𝟽)),('𝚝𝚘', (𝟾, 𝟷0)),('𝚝𝚑𝚎', (𝟷𝟷, 𝟷𝟺)),('🤗', (𝟷𝟻, 𝟷𝟼)),('𝚃𝚘𝚔𝚎𝚗𝚒𝚣𝚎𝚛𝚜', (𝟷𝟽, 𝟸𝟽))]

5/ 𝚖𝚘𝚍𝚎𝚕:
[('𝚆𝚎𝚕𝚌𝚘𝚖𝚎', (0, 𝟽)),('𝚝𝚘', (𝟾, 𝟷0)),('𝚝𝚑𝚎', (𝟷𝟷, 𝟷𝟺)),('🤗', (𝟷𝟻, 𝟷𝟼)),('𝚃𝚘𝚔𝚎𝚗𝚒𝚣𝚎𝚛𝚜', (𝟷𝟽, 𝟸𝟽))] ➜ [𝚠𝚎𝚕𝚌𝚘𝚖𝚎, 𝚝𝚘, 𝚝𝚑𝚎, [𝚄𝙽𝙺], 𝚝𝚘𝚔𝚎𝚗, ##𝚒𝚣𝚎𝚛, ##𝚜]

6/ 𝚙𝚘𝚜𝚝-𝚙𝚛𝚘𝚌𝚎𝚜𝚜𝚒𝚗𝚐:
[𝚠𝚎𝚕𝚌𝚘𝚖𝚎, 𝚝𝚘, 𝚝𝚑𝚎, [𝚄𝙽𝙺], 𝚝𝚘𝚔𝚎𝚗, ##𝚒𝚣𝚎𝚛, ##𝚜] ➜ [[𝙲𝙻𝚂], 𝚠𝚎𝚕𝚌𝚘𝚖𝚎, 𝚝𝚘, 𝚝𝚑𝚎, [𝚄𝙽𝙺], 𝚝𝚘𝚔𝚎𝚗, ##𝚒𝚣𝚎𝚛, ##𝚜, [𝚂𝙴𝙿]]

* notice [𝙲𝙻𝚂] $𝙰 [𝚂𝙴𝙿]

7/ tokens to ids conversion:
[[𝙲𝙻𝚂], 𝚠𝚎𝚕𝚌𝚘𝚖𝚎, 𝚝𝚘, 𝚝𝚑𝚎, [𝚄𝙽𝙺], 𝚝𝚘𝚔𝚎𝚗, ##𝚒𝚣𝚎𝚛, ##𝚜, [𝚂𝙴𝙿]] ➜ [𝟷0𝟷, 𝟼𝟷𝟼0, 𝟸000, 𝟷𝟿𝟿𝟼, 𝟷00, 𝟷𝟿𝟸0𝟺, 𝟷𝟽𝟼𝟸𝟿, 𝟸0𝟷𝟻, 𝟷0𝟸]

8/ Checkout 🤗 Tokenizers docs (w/ new look) huggingface.co/docs/tokenizer…

9/ 🤗 Course has an excellent section on tokenization by @LucileSaulnier for anyone who wants to learn more huggingface.co/course/chapter6