,
8 tweets,
2 min read
Read on Twitter
Speech synthesis as a field has an evaluation problem. The commonly used Mean Opinion Score (MOS) reported across various papers are not comparable. To make things worse, Deepvoice (from Baidu) papers report lower MOS of systems from Google (WaveNet, Tacotron, ..). It's a mess!
I am fairly confident, we as a field (except possibly a select few who know this experientially), do not know where we stand today with speech synthesis. There is absolutely no way you can look at two papers and conclude one is superior than the other.
Speech Synthesis is an art masquerading as science.
Even if you query these oracles (the handful of synthesis experts), it will not be useful. A Tacotron implementation at Google is not the Tacotron at Baidu, and most definitely not the one you find in open source or your startup.
This is even if you train all those systems on the same data. Because engineering and pre-processing artifacts, that get ignored or under-communicated, matter a lot.
People have so far used "look how cute my baby is" approach to "sell" synthesis work by shipping a page full of audio samples. It is impossible for any human to listen to these samples & give an opinion reliably. 
To make things worse, the more you listen, the more variance there is in the opinion. The longer you do the evaluation experiments, human errors kick in making it further unreliable.
With so many sources of variability, the MOS score is pretty much garbage.
