For those in the language modeling space, a question regarding perplexity as a metric with varying tokenization:
- Is there a hard proof showing for a dataset D being tokenized using A and B that the perplexity is equivalent?
- Does that proof take into account teacher forcing?
- Is there a hard proof showing for a dataset D being tokenized using A and B that the perplexity is equivalent?
- Does that proof take into account teacher forcing?
I ask as I have never seen a proof and always assumed smarter people than myself had thought about it. Intuitively I felt it reasonable until I recently began pondering over the teacher forcing aspect which is essentially giving your model supervision, including at test time.
Imagine you had the task of language modeling:
"Bob and Alice were fighting for first place but who won? [predict: Bob or Alice]"
The claim is that the language model's perplexity (confusion) should be equal regardless of how we split the text.
"Bob and Alice were fighting for first place but who won? [predict: Bob or Alice]"
The claim is that the language model's perplexity (confusion) should be equal regardless of how we split the text.
If we tokenize to "Bob" and "Alice" (words) then we get a far larger error in prediction than if we tokenize to "B|ob" and "A|lice" (wordpieces). The teacher's "supervision" at test time gives "A" or "B", collapsing the entropy, so you get the rest correct ("ob|lice").
Standard ambiguities too i.e. "special|ized" vs "special|ised". The dynamics of the neural network would certainly change - larger bursts of entropy more infrequently with words - but I don't know if that constitutes an unfair or unreasonable change in the model's operation.
- From a compression point of view this is reasonable as you can guarantee the next token is correct.
- From a language modeling point of view I'm less certain. It's Ouroboros - you're feeding the output of your model back as input. Losing the oracle teacher is big.
- From a language modeling point of view I'm less certain. It's Ouroboros - you're feeding the output of your model back as input. Losing the oracle teacher is big.
"The tokenization Ouroboros issue" seems to definitely have some manner of impact on how well perplexity as a metric will translate to real world generation, assuming a word vs wordpiece LM with equal perplexity, but perhaps that's outside of the scope of perplexity as a metric.
So, dear reader, have you pondered such things? Do you have an answer? My SotA claim depends on it! Most importantly however: the creation of future datasets. If all tokenizations are equal, all datasets are blobs. Otherwise they must remain specifically tokenized sequences.
