Read on Twitter
Sub 600ms latency speech conversational AI is completely possible today, surprised I haven’t seen anyone that does this.
The key is hosting a model (like llama), streaming from whisper, and every few tokens, prefilling more of the kv cache - without evicting from memory (1/4)
The key is hosting a model (like llama), streaming from whisper, and every few tokens, prefilling more of the kv cache - without evicting from memory (1/4)
Whisper could transcribe the remainder of the text after speaking in 100ms.
This means the time to first voice response is around the time to process the last several tokens of input, then generate the first 20 tokens, then pass that into a text to speech model. (2/4))
This means the time to first voice response is around the time to process the last several tokens of input, then generate the first 20 tokens, then pass that into a text to speech model. (2/4))
With quantized llama 70b using 2-way tensor parallelism, you could process + generate those tokens in 0.4s on two A100s (3/4)
The last step is getting a text to speech model to start talking within 100ms.
Again, this feels quite achievable if hosting a model and streaming the text into the models kv cache.
But I’m less confident on the state of text to speech oss models (4/4)
Again, this feels quite achievable if hosting a model and streaming the text into the models kv cache.
But I’m less confident on the state of text to speech oss models (4/4)
• • •
Missing some Tweet in this thread? You can try to
force a refresh
